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Abstract 


Defence  Research  and  Development  Canada  Toronto  Research  Centre  is  developing  the 
Environment  Familiarization  and  Indicator  Trainer  (EFIT)  as  part  of  the  Improvised  Explosive 
Device  (IED)  Awareness  Training  Project  of  the  Counter  Improvised  Explosive  Device 
Technology  Demonstration  Program.  EFIT  is  a  training  tool  that  provides  instruction  on  IED 
combat  indicators  and  what  to  look  for  in  a  given  area  of  operation. 

A  user  evaluation  was  conducted  with  a  prototype  of  Interactive  EFIT  software  with  seven 
members  of  the  Peace  Support  Training  Centre  in  Kingston  Ontario.  The  intent  of  the  evaluation 
was  to  determine  the  usability  of  Interactive  EFIT  and  to  gain  user  feedback  on  EFIT’s  training 
potential. 

Participants  were  briefed  on  how  to  use  the  software  and  they  were  asked  to  perform  four  tasks 
using  the  think-aloud  technique.  Observations  were  made  as  to  how  participants  interacted  with 
the  software.  Participants  also  completed  a  usability  questionnaire,  rated  the  usefulness  of 
Interactive  EFIT  for  seven  learning  objectives  and  rated  the  usefulness  of  proposed  features. 

Overall,  participants  evaluated  the  Interactive  EFIT  positively  and  found  it  to  have  potential  as  a 
training  tool.  Improvements  should  be  made  to  Interactive  EFIT  to  make  the  software  more 
flexible,  stimulating  and  intuitive. 


Significance  to  defence  and  security 


IEDs  posed  a  significant  threat  to  the  safety  of  Canadian  Armed  Forces  personnel  in  Afghanistan 
and  may  pose  threats  in  future  combat  missions.  Participants  in  this  study  evaluated  Interactive 
EFIT  positively  and  found  it  to  have  potential  to  train  the  detection  of  IED  combat  indicators  and 
found  it  useful  for  route  and  theatre-specific  terrain  familiarization.  Interactive  EFIT  can  be 
valuable  for  Theatre  Mission  Specific  Training  trainers  briefing  IED  threats.  Soldiers  could  also 
benefit  from  using  Interactive  EFIT  on  their  own  to  explore  routes  within  their  area  of  operations 
and  become  familiar  with  local  threats. 
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Resume 


Le  Centre  de  recherche  de  Recherche  et  developpement  pour  la  defense  Canada  -  Toronto 
travaille  a  la  mise  au  point  d’un  simulateur  d’indices  et  de  familiarisation  avec  Tenvironnement 
(EFIT)  dans  le  cadre  du  projet  de  sensibilisation  aux  dispositifs  explosifs  de  circonstance  (IED) 
du  Programme  de  demonstration  de  la  technologie  pour  contrer  les  dispositifs  explosifs  de 
circonstance.  L’EFIT  est  un  outil  offrant  une  formation  sur  les  indices  de  combat  avec  IED  et 
expliquant  ce  qu’il  faut  tenter  de  reperer  dans  une  zone  d’ operations  donnee. 

Sept  membres  du  Centre  de  formation  pour  le  soutien  de  la  paix  de  Kingston  (Ontario)  ont  evalue 
un  prototype  du  logiciel  interactif  EFIT.  Cette  evaluation  par  des  utilisateurs  visait  a  determiner  la 
convivialite  du  logiciel  et  a  obtenir  des  commentaires  sur  son  potentiel  comme  outil  de  formation. 

Les  participants  ont  ete  informes  sur  la  fagon  d’utiliser  le  logiciel  et  ont  ete  invites  a  accomplir 
quatre  taches  au  moyen  de  la  technique  de  reflexion  a  haute  voix.  On  a  ainsi  pu  observer 
comment  les  participants  utilisaient  le  logiciel.  Ceux-ci  ont  egalement  rempli  un  questionnaire  sur 
la  convivialite,  evalue  Tutilite  de  l’EFIT  interactif  relativement  a  sept  objectifs  d’apprentissage  et 
evalue  Tutilite  des  fonctionnalites  proposees. 

Dans  l’ensemble,  les  participants  ont  fait  une  evaluation  positive  du  simulateur  interactif  et 
estiment  qu’il  pourrait  constituer  un  bon  outil  de  formation.  Des  ameliorations  devraient  etre 
apportees  pour  creer  un  logiciel  plus  souple,  plus  stimulant  et  plus  intuitif. 


Importance  pour  la  defense  et  la  securite 


Les  IED  ont  represente  une  menace  importante  pour  la  securite  du  personnel  des  Forces  armees 
canadiennes  en  Afghanistan  et  sont  susceptibles  de  constituer  une  menace  lors  de  futures  missions 
de  combat.  Les  participants  a  cette  etude  ont  fait  une  evaluation  positive  de  l’EFIT  interactif  et 
estiment  qu’il  pourrait  servir  d’outil  de  formation  a  la  detection  d’indices  d’lED  en  situation  de 
combat;  ils  le  trouvent  utile  pour  se  familiariser  aux  routes  et  a  Tenvironnement  propre  a  un 
theatre.  Le  simulateur  interactif  pourrait  etre  utile  aux  instructeurs  offrant  l’instruction  axee  sur  le 
theatre  et  sur  la  mission  (IATM)  pour  exposer  les  menaces  liees  aux  IED.  Les  soldats  pourraient 
egalement  tirer  profit  de  l’EFIT  en  l’utilisant  pour  etudier  les  routes  dans  leur  zone  d’operations 
et  se  familiariser  avec  les  menaces  locales. 
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1  Introduction 


1.1  Background 

Improvised  Explosive  Devices  (IEDs)  posed  a  significant  threat  to  the  safety  of  Canadian  Armed 
Forces  (CAF)  personnel  deployed  in  Afghanistan  combat  missions.  In  2008  the  Chief  of  Defence 
Staff  stated  that  “[IEDs]  remain  the  single  most  dangerous  weapon  employed  by  insurgents  in 
Afghanistan,  accounting  for  the  majority  of  Canadian  combat  related  casualties”  (Chief  of 
Defence  Staff,  2008).  Despite  Canada’s  withdrawal  from  combat  operations  in  Afghanistan  it  is 
anticipated  that  soldiers  will  continue  to  face  the  threat  of  IEDs  in  future  missions.  Research  and 
development  efforts  continue  throughout  Defence  Research  and  Development  Canada  (DRDC) 
and  the  CAF  to  mitigate  IED  casualties. 

Many  counter-IED  efforts  focus  on  developing  new  technologies  to  detect,  avoid,  neutralize  or 
protect  personnel  from  IEDs.  However,  despite  technological  advances,  detecting  IEDs  still 
depends  very  much  on  the  judgment  and  unaided  perceptual  skills  of  soldiers  (Zorpette,  2008). 
Accordingly,  the  CAF  Counter-IED  Tactics,  Techniques  and  Procedures  (TTPs)  handbook 
(Department  of  National  Defence,  2006)  directs  personnel  to  monitor  their  environment  for  the 
presence  of  a  number  of  perceptual  cues  (“combat  indicators”)  that  suggest  an  elevated 
probability  of  the  presence  of  IEDs.  These  indicators,  such  as,  abnormal  pattern  of  life,  freshly 
dug  holes,  new  asphalt  patches,  canalizing  terrain,  IED  components  (fuse  and  firing  mechanism, 
detonator,  explosive,  casing,  detonator  power  source),  altered  vehicles,  etc.  and  the  task  of 
searching  for  them  are  typically  taught  during  pre-deployment  Theatre  Mission  Specific  Training 
(TMST)  through  PowerPoint  theory  lessons,  “petting  zoos”  of  reproduced  or  actual  IEDs 
retrieved  from  theatre,  and  walk-throughs  of  IED  lanes. 

The  IED  Awareness  Training  Project  at  DRDC  Toronto,  part  of  the  Counter  Improvised 
Explosive  Device  Technology  Demonstration  Program  (Counter-IED  TDP)  looked  at  ways  of 
improving  training  of  IED  combat  indicators.  Keillor,  Jarmasz,  Pavlovic  and  Lamb  (2007) 
showed  that  troops  with  Afghanistan  road  travel  experience  scan  for  threats  in  video  from  a 
moving  vehicle  on  Afghanistan  roads  differently  than  civilians  who  hadn’t  been  to  Afghanistan. 
This  suggests  that  there  are  trainable  skills  for  visually  scanning  real  scenery  that  may  improve 
IED  detection. 

To  improve  the  training  of  visual  scanning  for  IEDs  we  developed  various  prototypes  of  the 
Environment  Familiarization  and  Indicator  Trainer  (EFIT).  EFIT  combined  real  video  of  convoy 
operations  with  a  map  of  high  resolution  satellite  imagery  and  vector  data  (roads,  culverts, 
locations  of  past  IED  incidents,  and  other  terrain  features).  The  videos  were  accompanied  by 
subject  matter  expert  (SME)  commentary  and  visual  overlays  to  direct  the  attention  of  the  viewers 
to  IED  combat  indicators.  In  an  interactive  prototype  of  EFIT  users  could  pan  and  zoom  on  the 
map,  launch  moving  vehicle  videos,  and  obtain  more  information  about  previous  IED  incidents. 

1 .2  Study  objective 

The  primary  objective  of  this  study  was  to  determine  how  usable  Interactive  EFIT  was  for 
soldiers  required  to  learn  about  the  Afghanistan  environment  and  IED  combat  indicators.  By 


DRDC-RDDC-201 5-R009 


1 


usable  it  is  generally  meant  that  software  is  “easy  to  learn,  effective  to  use,  and  provide  an 
enjoyable  user  experience”  (Preece  et  al.,  2002). 

The  secondary  objective  of  this  study  was  to  gain  user  feedback  on  EFIT’s  training  potential1. 
EFIT  designers  expected  EFIT  to  reduce  the  amount  of  time  in  theatre  becoming  familiar  with 
commonly  travelled  routes  and  the  cultural  and  natural  environment  of  the  CAF  area  of 
operations  (AO).  Troops  interviewed  by  Keillor  et  al.  (2007)  reported  developing  an  intuitive 
sense  for  situations  of  high  threat  or  those  with  a  high  potential  for  IEDs  after  a  few  months  into 
their  first  rotation  in  theatre.  Often  this  “spidey  sense”  or  pattern  knowledge  was  difficult  to 
verbalize  into  a  clear  set  of  rules  or  cues.  EFIT  designers  expected  that  through  “training  the 
background”,  or  training  what  the  normal  day-to-day  Afghanistan  environment  (or  other  theatres 
of  operation)  looks  like  through  watching  video  from  a  moving  vehicle  alongside  a  dynamic  map, 
EFIT  could  help  improve  troops’  ability  to  recognize  abnormal  situations.  EFIT  designers 
anticipated  that  by  passing  knowledge  and  lessons  learned  from  previous  rotations,  EFIT  would 
provide  familiarization  of  common  IED  threat  types,  introduce  common  IED  combat  indicators 
by  threat  type  in  visual  context,  and  train  possible  Counter-IED  actions  to  mitigate  threat.  This 
study  sought  SME  feedback  on  whether  EFIT  could  be  a  useful  training  tool  in  this  regard. 

The  results  of  this  study  will  inform  future  EFIT  design  decisions  ensuring  the  end  product  meets 
the  needs  of  its  CAF  users. 

1.3  EFIT  development 

1.3.1  Early  prototypes 

The  first  prototype  of  the  EFIT  concept  was  developed  in  February  2007.  It  consisted  of  a  one 
minute  video  clip  from  a  ground  vehicle  in  Afghanistan  alongside  an  aerial  photo  with  a  moving 
track  showing  the  position  of  the  video  camera.  A  screenshot  of  the  video  is  shown  in  Annex  A 
Figure  A.l.  The  concept  was  further  developed  with  a  second  video  prototype  shown  in  Annex  A 
Figure  A.2  that  built  on  the  original  idea,  but  added  overlays  onto  the  video  that  could  be  used  to 
point  out  IED  combat  indicators  or  points  of  interest.  The  map  portion  of  the  video  was  also 
augmented  to  increase  the  visibility  of  the  track  of  the  camera,  show  points  of  interest  within  the 
environment,  and  demonstrate  how  future  versions  of  the  concept  could  have  an  interactive  map 
(panning  and  zooming). 

1.3.2  Non-Interactive  EFIT 

Two  further  prototypes  of  EFIT  (non-interactive  EFIT)  were  completed  in  January  2008.  SMEs, 
mostly  Combat  Engineers  with  Afghanistan  deployment  experience  from  a  local  Reserve  unit, 
reviewed  video  footage  of  Kandahar  province  and  noted  IED  combat  indicators.  Multiple  SMEs 
reviewed  each  segment  of  video.  SME  comments  were  pooled  and  reviewed  for  how  often  they 
identified  the  same  IED  combat  indicators.  Clips  were  selected  based  on  the  diversity  of  terrain, 


1  A  scientific  training  effectiveness  assessment  of  more  recent  versions  of  EFIT  can  be  found  in  Zotov  et  al. 
(2014).  Zotov  et  al.  showed  participants  can  improve  their  IED  threat  assessment  skills  by  watching  EFIT 
videos. 
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built  environment,  and  the  quantity  of  multiply  identified  IED  combat  indicators.  An 
approximately  1.5  minute  rural  clip  between  Kandahar  Airfield  (KAF)  and  Kandahar  City 
(Figure  1)  and  an  approximately  two  minute  urban  clip  within  Kandahar  City  (Figure  2)  were 
selected.  A  script  for  each  video  was  written  using  the  SME  notes.  The  scripts  pointed  out  IED 
combat  indicators,  points  of  interest,  and  possible  tactical  actions.  The  scripts  were  read, 
recorded,  and  added  as  audio  tracks  to  the  non-interactive  EFIT  prototypes.  On  the  right  side  of 
each  video  was  an  image  from  70  cm  resolution  imagery  (one  pixel  for  every  70  cm  in  the  real 
world).  A  red  line  indicated  the  route  the  ground  vehicle  traveled. 


Figure  1:  Screenshot  of  rural  non-interactive  EFIT  prototype  between  Kandahar  Airfield  and 

Kandahar  City. 


Figure  2:  Screenshot  of  urban  non-interactive  EFIT  prototype  within  Kandahar  City. 


Throughout  the  development  of  EFIT  the  various  prototypes  underwent  a  series  of  usability 
evaluations  consistent  with  the  principles  of  interaction  design.  Interaction  design  is  a  discipline 
which  provides  guidelines  on  how  to  optimally  design  software  interfaces  to  suit  the  environment 
in  which  they  are  to  operate.  Interaction  design  dictates  that  “identifying  usability  and  user 
experience  goals  is  essential  for  making  every  product  successful”  (Preece  et  al.,  2002)  and  that 
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an  understanding  of  users’  needs  should  be  reflected  in  the  design  of  the  software 
(Preece  et  ah,  2002). 

The  2nd  Combat  Engineer  Regiment  (2CER),  in  Petawawa,  used  the  two  non-interactive  EFIT 
prototypes  as  part  of  Explosive  Threat  and  Hazard  Awareness  and  Recognition  (ETHAR)  lectures 
during  TMST  for  Task  Force  (TF)  3-08  (Operation  Athena  Rotation  6).  1CER  used  the  two 
non-interactive  prototypes  as  part  of  TF  3-09  (Operation  Athena  Rotation  8)  training. 

Eleven  training  instructors  and  81  students  (32  from  TF  3-08  and  49  from  TF  3-09)  provided 
feedback  on  the  prototypes  by  completing  a  brief  Usefulness  Questionnaire  (Annex  C.7).  Full 
Usefulness  Questionnaire  results  are  presented  in  Section  3.2.1,  but  one  of  the  key  results  from 
the  non-interactive  EFIT  TF  3-08  testing  was  that  instructors  felt  EFIT  could  be  improved  as  an 
instructional  tool  by  making  the  map  interactive.  Instructors  felt  an  interactive  map  with  zoom 
would  be  useful  to  demonstrate  map  reconnaissance  (recce)  prior  to  watching  the  video. 

1.3.3  Interactive  EFIT 

A  paper  prototype  (non-functional  PowerPoint  interface  sketch)  for  an  interactive  version  of  EFIT 
(Figure  A.3)  was  drafted  in  mid-2008.  Contained  within  a  standard  Windows  graphical  user 
interface  the  prototype  illustrated  an  interactive  map  with  panning  and  zooming,  symbols 
representing  past  IED  incidents,  options  for  switching  background  imagery  (satellite,  map,  etc.), 
the  ability  to  right-click  and  select  from  multiple  videos  of  the  same  route  taken  on  different 
dates/times  of  day,  position  in  vehicle,  or  type  of  sensor.  There  were  also  controls  for  video 
playback,  visual  overlays  of  IED  combat  indicators,  and  soundtrack  options  for  selecting  between 
SME  commentary,  real  radio  communications  (Comms),  etc. 

The  paper  prototype  was  partially  implemented  as  an  add-on  to  ArcGIS  9.2  (Esri,  Redlands, 
California),  a  commercial  mapping  and  data  analysis  software  using  Visual  Basic  for 
Applications  (Microsoft  Corporation,  Redmond,  Washington).  A  screenshot  of  the  Interactive 
EFIT  prototype  evaluated  in  this  study  is  shown  in  Figure  3.  Interactive  EFIT  consisted  of  four 
main  interface  components:  1.  interactive  map,  2.  route  video  panel,  3.  options  panel,  and  4.  EFIT 
toolbar.  The  interactive  map  shown  in  Figure  4  could  be  used  to  view  roads,  past  IED  incidents 
and  select  route  video  tracks  to  watch.  Roads  were  shown  on  the  map  using  green  lines,  past  IED 
incidents  were  shown  with  diamond  symbols  and  labelled  with  the  type  of  IED.  The  route  video 
tracks  were  shown  by  thick  blue  lines  augmented  with  arrows  to  show  the  video’s  forward 
direction.  Tracks  could  be  clicked  to  play  video  of  the  track  at  the  location  clicked  (the  tracks 
were  made  up  of  a  series  of  selectable  line  segments  although  they  appeared  continuous  to  users). 
The  route  video  panel  allowed  users  to  play,  stop,  pause  and  fast  forward  through  the  route  video 
tracks. 
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EFIT  Toolbar 


Figure  3:  Screenshot  of  the  interactive  EFIT  prototype  used  in  this  study.  Note  the  four  main 
interface  components:  1.  options  panel,  2.  route  video  panel,  3.  interactive  map, 

and  4.  EFIT  toolbar. 


Route  Video  Tracks 


Past  lEDs 


Roads 


Figure  4:  Close-up  of  interactive  map. 

The  options  panel  listed  all  of  the  route  video  tracks  that  are  loaded  into  Interactive  EFIT.  To  play 
a  video,  users  clicked  on  the  name  of  the  route  and  clicked  the  play  icon  on  the  route  video  panel. 
The  options  panel  also  had  options  to  select  whether  you  want  to  play  a  familiarization  video  of 
the  route  video  or  an  indicator  training  video.  The  familiarization  video  was  an  unedited  video 
whereas  the  indicator  training  video  had  expert  commentary  about  the  route  and  highlighted 
potential  IED  combat  indicators.  The  tools  required  to  operate  Interactive  EFIT  were  located  on 
the  EFIT  toolbar  shown  in  Figure  5. 

Figure  5:  EFIT  toolbar. 
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Where: 


^  Opened  the  Interactive  EFIT  program. 

k  Allowed  users  to  select  and  click  on  the  map.  While  watching  a  video  users  could  click  on 
"'d  part  of  the  track  to  jump  to  that  location  in  the  video. 


$ 


Allowed  users  to  open  incident  reports  linked  to  a  particular  IED  strike.  An  IED  strike 
location  turned  blue  when  rolled  over  with  the  mouse. 

Allowed  users  to  pan  the  map.  Clicking  and  holding  with  the  middle  mouse  button 
panned  the  map. 


Allowed  users  to  zoom  in  on  the  map.  The  middle  scroll  wheel  could  also  be  used. 


Allowed  users  to  zoom  out  on  the  map.  The  middle  scroll  wheel  could  also  be  used. 


In  order  to  evaluate  whether  Interactive  EFIT  supported  user  needs  and,  thus,  was  suitable  for 
eventual  integration  in  pre-deployment  training,  user  testing  of  the  software  was  required.  This 
report  documents  the  user  testing  of  Interactive  EFIT  with  SMEs  at  the  Peace  Support  Training 
Centre. 
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2  Methods 


2.1  Study  overview 

This  study  was  designed  to  evaluate  the  usability  of  Interactive  EFIT  by  having  CAF  personnel 
perform  four  tasks  with  the  system.  The  tasks  were  based  on  previous  discussions  with  SMEs  as 
to  what  TMST  students  and  instructors  might  typically  do  when  the  final  version  of  EFIT  is 
delivered.  We  recorded  participants’  interactions  with  the  prototype  as  they  completed  the  tasks. 
Participants  were  encouraged  to  speak  aloud,  describe  what  they  were  doing  and  what  they 
expected  of  the  prototype  as  they  completed  their  tasks.  These  comments  were  recorded  along 
with  the  rates  of  feature  use  and  “errors”  (situations  where  the  designers’  expectations  of  how 
users  would  use  Interactive  EFIT  did  not  correlate  with  the  participant’s  actions).  Following 
completion  of  the  tasks,  participants  completed  a  usability  questionnaire  and  a  usefulness 
questionnaire.  The  remainder  of  this  chapter  provides  further  details  of  these  methods. 

2.2  Participants 

Seven  male  participants  were  recruited  from  the  Peace  Support  Training  Centre  (PSTC)  at 
Canadian  Forces  Base  (CFB)  Kingston  March  10-12  2009.  All  participants  met  the  standard 
personnel  requirements  for  the  CAF  (active  duty  age  range  18  to  60)  with  standard  vision  and 
health.  Years  of  service  ranged  from  12  to  25  years.  A  Major,  Captain,  Master  Warrant  Officer, 
two  Warrant  Officers  and  two  Sergeants  participated.  Participants  had  attended  Mine  Awareness 
Training  (MAT)  or  ETHAR  between  three  and  over  20  times  in  their  careers.  All  but  one 
participant  had  deployment  experience  in  Afghanistan.  Participants  were  not  remunerated  for 
their  participation  as  no  abnormal  stress  or  discomfort  was  anticipated  (DAOD  5061-1,  1998). 

2.3  Room  set-up 

The  user  evaluations  took  place  over  three  days,  March  10-12,  2009  in  a  meeting  room  at 
CFB  Kingston.  Interactive  EFIT  was  setup  in  the  centre  of  the  table  on  a  computer  with  an 
18  inches  liquid  crystal  display  monitor  (see  Figure  6).  A  laptop  was  placed  to  the  right  of  the 
main  computer  with  a  QuickCam  pro  for  Notebooks  (Fogitech,  Silicon  Valley,  California) 
webcam  to  record  the  participant  sessions.  Participants  sat  in  a  chair  approximately  60cm  in  front 
of  the  Interactive  EFIT  computer.  Data  recorders  sat  on  either  side  of  the  participant  (Figure  7). 
Participants  were  free  to  adjust  the  chair  to  a  comfortable  seating  position. 
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Figure  6:  User  evaluation  room  set-up.  Interactive  EFIT  computer  in  center  with  video  recording 

laptop  on  right. 


Figure  7:  A  participant  using  Interactive  EFIT.  Note  how  this  participant  has  maximized  the  map 

to  fill  the  whole  screen. 


2.4  Interactive  EFIT  set-up 

The  Interactive  EFIT  prototype  was  loaded  with  three  route  videos,  each  with  an  in-vehicle  audio 
track  and  two  with  expert  commentary  and  visual  overlays.  The  first  video  was  a  one  minute, 
35  second  long  route  between  KAF  and  Kandahar  City  (Figure  8a).  The  expert  commentary 
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describing  potential  aiming  markers,  culvert  concealment,  canalized  situations  and  civilian 
behaviour  can  be  found  in  Annex  B.l.  Items  described  in  the  audio  commentary  were  highlighted 
with  black  circle  outlines. 


The  second  video  was  a  two  minute,  20  second  long  route  within  Kandahar  City  departing  from 
the  Governor’s  Compound  (Figure  8b).  The  expert  commentary  describing  chokepoints,  civilian 
behaviour,  suicide  vehicle-borne  IEDs  and  firing  points  can  be  found  in  Annex  B.2.  The 
commentary  version  of  this  video  also  had  black  circle  outlines.  The  third  video  was  a  14  minute, 
38  second  long  route  from  Ma'Sum  Ghar  (MSG)  to  Patrol  Base  Sperwan  Ghar  (PBSG) 
(Figure  8c).  The  third  video  had  an  audio  track  of  voice  actors  simulating  the  communications 
between  a  crew  commander,  driver,  gunner  and  convoy  commander.  The  Comms  script 
(Annex  B.3)  was  derived  from  observations  SMEs  had  made  when  reviewing  the  video.  The 
Comms  also  contained  informal  chatter  between  vehicle  members. 


(a)  (b)  (c) 


Figure  8:  Screen  shots  from  the  three  trial  videos:  (a)  paved  rural,  (b)  paved  urban,  (c)  dirt  rural. 


2.5  Procedure 

Participants  were  greeted  and  asked  to  read  the  information  sheet  (Annex  C.3)  and  consent  form 
(Annex  C.4).  They  were  given  a  chance  to  ask  any  questions  pertaining  to  the  study  before 
signing  the  consent  form  if  they  wished  to  participate.  Participants  were  then  given  an  evaluation 
package  containing  a  description  of  Interactive  EFIT,  its  interface  components,  the  tasks  they 
would  be  asked  to  perform  (Annex  C.5),  a  Usability  Questionnaire  derived  from  the  work  of 
Chin  et  al.  (1988,  Annex  C.6),  and  a  Usefulness  Questionnaire  (Annex  C.7).  The  usability  items 
focused  on  the  participants’  interaction  experience  with  the  software  interface  whereas  the 
usefulness  items  asked  participants  about  new  features  and  how  effective  the  software  was  as  a 
trainer.  As  participants  worked  through  the  evaluation  package  they  were  asked  if  they  had  any 
questions  to  ensure  they  understood  the  experimental  protocol. 

Interface  images  and  text  descriptions  in  the  Interactive  EFIT  101  section  of  the  evaluation 
package  (Annex  C.5)  were  used  to  introduce  participants  to  the  map  symbology  for  the  route 
video  tracks,  IEDs  and  roads.  The  evaluation  package  informed  participants  that  clicking  on  the 
map  tracks  plays  video  tracks.  The  evaluation  package  described:  1)  the  route  video  panel  where 
participants  could  stop,  pause,  play  or  fast  forward  through  the  route  video  tracks,  2)  the  options 
panel  where  participants  play  route  videos  with  and  without  expert  commentary,  and  3)  the  EFIT 
toolbar  with  the  tools  to  open  the  Interactive  EFIT  program,  select  a  video  on  the  map,  open  IED 
incident  reports,  pan  the  map,  and  zoom  in  and  out  on  the  map. 
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Once  participants  felt  they  were  comfortable  with  the  Interactive  EFIT  interface  components  they 
read  the  instructions  for  the  first  task  and  began  using  Interactive  EFIT.  The  four  tasks  were 
potential  use  cases  for  Interactive  EFIT  and  were  created  to  encourage  participants  to  use  the 
various  features  of  the  Interactive  EFIT  prototype.  The  instructions  given  for  each  task  are  as 
follows: 

1.  You  will  be  deployed  to  Afghanistan  in  several  months  and  have  received  EFIT  as  part  of 
your  pre-deployment  training  package.  You  know  that  you  will  be  travelling  between 
Kandahar  Air  Field  (KAF)  and  Kandahar  City  frequently.  To  prepare  for  your  tour,  take  some 
time  to  get  used  to  what  the  route  looks  like.  It  is  expected  that  you  will  be  in  the  gunner 
position  for  the  majority  of  your  tour. 

2.  The  IED  threat  in  Afghanistan  is  currently  high.  To  prepare  yourself  for  this  high  level  of 
threat,  learn  about  potential  IED  indicators  on  the  route  between  KAF  and  Kandahar  City. 

3.  You  are  about  to  go  on  your  first  convoy  mission  through  Kandahar  City  and  you  want  to 
learn  more  about  the  types  of  IED  indicators  that  typically  occur  in  an  urban  environment. 
Your  crew  commander  tells  you  that  canalizing  from  vehicles  [sic]  is  something  that  you  will 
need  to  prepare  for.  List  below  examples  of  canalization  on  the  route  through  Kandahar  City. 

4.  You  have  been  tasked  to  be  an  air  sentry  on  a  convoy  travelling  between  MSG  and  PBSB 
[sic]  and  you  want  to  be  aware  of  possible  IED  threats  on  that  route.  Find  out  what  the  most 
common  type  of  IED  used  on  this  route  was  for  2008.  List  below  the  key  messages  from  the 
crew  commander  in  the  instructional  video. 

While  the  participants  completed  these  tasks  they  were  asked  to  follow  the  “think-aloud” 
protocol.  Participants  were  encouraged  to  describe  their  actions,  intentions,  frustrations  and 
appreciations  as  they  used  the  software.  We  took  written  notes  of  the  participant’s  comments  and 
actions  in  addition  to  video  recording  the  sessions. 

Upon  completing  the  four  tasks  participants  were  asked  to  fill  out  the  27  item  Questionnaire  for 
User  Interface  Satisfaction  (QUIS)  (Chin  et  al.  1988,  Annex  C.6),  henceforth  referred  to  as  the 
usability  questionnaire.  The  usability  questionnaire  gauged  the  participant’s  overall  reaction  to 
the  software,  screen  attributes,  terminology  and  system  information,  learning,  system  capabilities, 
the  three  most  negative  aspects  of  the  software  and  the  three  most  positive  aspects.  Participants 
were  also  asked  to  fill  out  a  Usefulness  Questionnaire  (Annex  C.7)  regarding  their  background 
with  the  CAF,  to  rate  the  usefulness  of  15  proposed  additions/improvements  to  Interactive  EFIT, 
and  rate  the  usefulness  of  Interactive  EFIT  for  learning  seven  of  the  key  EFIT  learning  objectives 
(e.g.,  becoming  familiar  with  routes,  finding  IED  indicators,  etc.).  Participants  were  also  asked 
what  they  thought  they  learned  from  Interactive  EFIT  and  to  provide  any  additional  comments. 
The  entire  evaluation  took  one  to  two  hours  to  complete. 
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2.6 


Measures 


2.6.1  Usability 

Qualitative  feedback  was  the  primary  measure  to  evaluate  the  usability  of  Interactive  EFIT. 
Participants’  verbal  comments  were  noted  as  well  as  the  difficulties  we  observed  as  participants 
attempted  to  complete  the  four  tasks.  Participants’  feedback  on  the  QUIS  Usability  Questionnaire 
as  to  the  three  most  negative  aspects  of  the  software  and  the  three  most  positive  aspects  also 
provided  an  indication  of  how  the  software  could  be  improved. 

The  QUIS  Usability  Questionnaire  also  provided  quantitative  measures  of  the  usability  of 
Interactive  EFIT.  The  QUIS  Usability  Questionnaire  had  27  ratings  with  a  10  point  Likert  scale 
from  zero  to  nine  from  left  to  right.  Each  rating  was  anchored  at  both  endpoints  with  adjectives 
(e.g.,  inconsistent/consistent).  Negative  adjectives  were  always  positioned  on  the  left  (0)  and 
positive  adjectives  were  positioned  on  the  right  (9).  Participants  also  had  the  option  to  select  Not 
Applicable  (NA)  for  each  rating.  In  a  previous  study  by  Chin  et  al.  (1988)  users  on  average  rated 
software  products  they  liked  higher  than  ones  they  disliked  for  all  27  ratings. 

The  seven  participants’  usability  questionnaire  data  were  averaged.  Those  questions  with  the 
highest  ratings  indicate  areas  where  the  software  is  on  the  right  track,  while  lower  scores  indicate 
areas  where  the  software  can  be  improved.  We  considered  ratings  from  zero  to  four  to  be  negative 
feedback  and  ratings  five  or  above  as  positive  feedback. 

We  also  recorded  whether  participants  used  software  features.  Low  rates  of  feature  use  across 
participants  indicate  that  a  feature  may  be  difficult  to  use,  or  require  further  training. 

2.6.2  EFIT’s  training  potential 

The  primary  mechanism  to  gauge  Interactive  EFIT’s  training  potential  was  the  Usefulness 
Questionnaire.  The  Usefulness  Questionnaire  had  five  questions.  The  first  question  asked 
participants  about  their  military  service  experience.  The  second  question  asked  participants  to  rate 
the  usefulness  of  15  proposed  additions/improvements  to  Interactive  EFIT  on  a  scale  from  one 
(not  at  all  useful)  to  five  (extremely  useful).  The  third  question  asked  participants  to  rate  the 
usefulness  of  Interactive  EFIT  to  learn  seven  specific  skills  on  a  scale  from  one  (not  at  all  useful) 
to  five  (extremely  useful).  The  fourth  question  asked  participants  if  they  learned  anything  new 
while  observing  Interactive  EFIT  that  was  not  included  in  the  training  they  had  previously 
received.  And  the  fifth  question  asked  participants  to  provide  any  additional  comments. 

We  also  captured  verbal  comments  on  how  Interactive  EFIT  could  be  used  beyond  its  envisioned 
use  by  ETHAR  trainers  briefing  TMST  IED  threats. 
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3  Results  and  discussion 


3.1  Usability 

3.1 .1  Usability  questionnaire 

Due  to  the  small  number  of  participants  we  did  not  perform  statistical  analyses  on  quantitative 
data.  The  usability  questionnaire  rating  frequencies  are  shown  in  Table  1.  Higher  ratings  indicate 
better  usability.  The  average  rating  across  participants  is  shown  in  the  rightmost  column.  The 
three  most  highly  rated  metrics  “Terminology  related  to  task”  and  “Performing  tasks  is 
straightforward”  were  rated  towards  “always”,  on  an  never/always  scale,  with  an  average  of 
7.7  and  “Correcting  your  mistakes”  was  considered  fairly  “easy”,  on  an  difficult/easy  scale,  with 
an  average  rating  of  7.6.  The  most  poorly  rated  metric  was  participant’s  overall  reaction  to  the 
software’s  flexibility  with  an  average  rating  of  5.7  nearly  halfway  between  “rigid”  and  “flexible.” 
Also  poorly  rated  were  the  software’s  ability  to  inspire  excitement  (average  rating  of  6  on  a  scale 
between  dull  and  stimulating)  and  its  ability  to  inform  users  of  its  progress  (average  rating  of  6  on 
a  scale  between  never  and  always). 
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Table  1:  Usability  questionnaire  rating  frequencies.  The  three  most  positive  aspects  are  highlighted  in  green  and  the  three  most  negative 

aspects  are  highlighted  in  orange. 


Overall  Reaction  to  the  software 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

NA 

Average 

1 

terrible 

1 

2 

4 

wonderful 

6.4 

2 

difficult 

2 

2 

3 

easy 

6.1 

3 

frustrating 

1 

3 

3 

satisfying 

6.3 

4 

inadequate  power 

1 

1 

2 

1 

2 

adequate  power 

6.3 

5 

dull 

1 

2 

2 

1 

1 

stimulating 

6 

6 

rigid 

1 

1 

2 

3 

flexible 

5.7 

Screen 

7  Reading  characters  on  the  screen 

hard 

1 

1 

4 

1 

easy 

6.7 

8  Highlighting  simplifies  task 

not  at  all 

1 

3 

1 

1 

very  much 

1 

6.3 

9  Organization  of  information 

confusing 

1 

4 

1 

very  clear 

1 

7 

10  Sequence  of  screens 

confusing 

1 

4 

1 

very  clear 

1 

7 

Terminology  and  system  information 

1 1  Use  of  terms  throughout  system 

inconsistent 

1 

3 

2 

consistent 

1 

7.2 

12  Terminology  related  to  task 

never 

3 

2 

1 

always 

1 

7.7 

13  Position  of  messages  on  screen 

inconsistent 

2 

1 

2 

consistent 

2 

7 

14  Prompts  for  input 

confusing 

1 

2 

1 

1 

clear 

2 

6.4 

15  Computer  informs  about  its  progress 

never 

1 

2 

1 

always 

3 

6 

16  Error  messages 

unhelpful 

2 

1 

1 

helpful 

3 

6.3 

Learning 

17  Learning  to  operate  the  system 

difficult 

1 

1 

2 

2 

1 

easy 

7.1 

18  Exploring  new  features  by  trial  and  error 

difficult 

1 

3 

2 

1 

easy 

7.3 

19  Remembering  names  and  use  of  commands 

difficult 

2 

1 

2 

1 

easy 

1 

7.3 

20  Performing  tasks  is  straightforward 

never 

3 

3 

1 

always 

7.7 

21  Help  messages  on  the  screen 

unhelpful 

1 

2 

helpful 

4 

6.3 

22  Supplemental  reference  materials 

confusing 

4 

clear 

3 

7 

System  Capabilities 

23  System  Speed 

too  slow 

1 

3 

2 

1 

fast  enough 

7.1 

24  System  reliability 

unreliable 

1 

1 

4 

1 

reliable 

6.1 

25  System  tends  to  be 

noisy 

2 

2 

1 

1 

quiet 

1 

7.2 

26  Correcting  your  mistakes 

difficult 

5 

2 

easy 

7.6 

27  Designed  for  all  levels  of  users 

never 

1 

1 

3 

2 

always 

7 
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Overall  the  results  from  the  questionnaire  were  positive.  Average  ratings  for  all  items  were 
greater  than  five,  indicating  participants  tended  towards  positive  feedback.  Participants  may  have 
considered  the  software  somewhat  inflexible  due  to  the  inability  to  resize  the  video  window.  Also 
the  trial  tasks  were  fairly  structured,  that  is,  we  wanted  participants  to  watch  specific  videos  for 
each  task.  Participants  were  free  to  explore  the  Interactive  EFIT  interface/map  for  each  task,  but 
there  were  specific  videos  we  wanted  participants  to  watch.  If  they  navigated  too  far  away  from 
the  video  we  wanted  them  to  watch  we  guided  them  to  the  area  containing  the  videos  we  wanted 
them  to  watch.  Participants  who  had  their  exploration  cut  short  may  have  felt  it  was  the  software 
that  was  inflexible. 

The  average  rating  of  6  on  the  dull/stimulating  scale  needs  to  be  considered.  As  EFIT  is  a  training 
tool  it  is  important  that  students  find  it  stimulating  otherwise  they  will  tune  out.  Participants  were 
excited  to  see  video  of  areas  that  they  had  travelled  on  a  previous  deployment,  yet  uninterested  in 
watching  video  that  was  outdated.  It  is  also  possible  that  the  video  training  lessons  were  second 
nature  for  participants  who  had  attended  MAT  or  ETHAR  over  68  times  in  their  careers.  See 
Section  3.2.1  for  a  discussion  of  what  participants  felt  they  learned  from  Interactive  EFIT. 

The  relatively  low  rating  of  the  software’s  ability  to  inform  users  of  its  progress  is  difficult  to 
explain.  Participants  may  not  have  had  a  clear  understanding  of  what  progress  meant  in  this 
context.  The  software  does  not  have  many  sequential  processes  where  user  input  is  required. 
Consequently,  three  participants  considered  the  question  “not  applicable.”  The  software  does 
have  a  progress  bar  below  the  video  to  show  how  far  along  the  video  has  elapsed.  The  map  also 
has  an  indicator  for  the  position  of  the  video  on  the  map.  When  a  video  is  played,  the  track  on  the 
map  turns  yellow  as  the  video  moves  forward  in  space.  We  neglected  to  describe  the  map 
progress  bar  in  the  instructions  prepared  for  participants.  If  participants  did  not  look  at  the  yellow 
map  bar  when  the  video  was  playing  they  would  not  have  been  able  to  interpret  what  the  blue  and 
yellow  meant  when  the  video  was  not  playing.  Further  instructions  describing  the  progress  bar  on 
the  map  are  required.  Interactive  EFIT  might  also  benefit  from  a  better  interface  integration  of  the 
progress  bar  under  the  video  and  the  progress  bar  on  the  map.  If  the  progress  bar  on  the  map  is 
designed  properly  then  the  progress  bar  under  the  video  becomes  redundant. 

Participants  may  also  have  interpreted  the  rating  “Computer  informs  about  its  progress”  to  mean 
that  the  software  would  inform  the  user  of  their  progress  using  it.  Since  Interactive  EFIT  is  a 
training  tool  participants  may  have  expected  that  the  software  would  provide  them  with  feedback 
as  to  how  well  they  progressing  through  the  trainer.  They  may  have  expected  feedback  such  as 
“you  have  reviewed  10  of  the  50  videos”  or  “you  have  read  five  of  the  10  IED  reports  for  this 
route.” 

When  listing  the  three  most  negative  aspects  of  the  prototype  four  of  the  seven  participants 
recommended  a  larger  screen.  The  preference  for  a  larger  screen  may  originate  with  participant’s 
experiences  viewing  video  and  maps  on  large  wall  mounted  displays.  Paper  maps  also  offer  a 
larger  display  than  the  18  inch  screen  that  was  used  for  the  trial.  Participants  may  also  have  been 
expressing  a  frustration  that  the  video  window  could  not  be  enlarged. 

The  second  most  reported  negative  aspect  of  the  prototype  listed  by  participants  (two  of  seven) 
was  the  audio.  This  can  be  elaborated  by  noting  that  the  expert  commentary  for  the  KAF  to 
Kandahar  City  video  was  very  low  volume,  participants  did  not  like  the  fact  that  the  crew  Comms 
for  the  MSG  to  PBSG  video  were  simulated,  the  Comms  for  the  MSG  to  PBSG  video  were 
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difficult  to  hear  over  the  vehicle  engine  noise  and  the  trial  room  was  very  noisy  from  what  might 
have  been  an  heating,  ventilation,  and  air  conditioning  (HVAC)  system  component. 

Participants  also  suggested  adding  route  names,  better  video  quality,  night  and  evening  videos, 
videos  from  different  positions  in  the  vehicle  and  panoramic  videos. 

For  the  three  most  positive  aspects  of  the  software,  four  participants  most  appreciated  the  video 
footage  and  the  training  potential  of  the  Interactive  EFIT  prototype.  Three  out  of  seven 
participants  listed  the  easy  to  use  and  navigable  interface  and  two  participants  listed  the  map. 

3.1 .2  Participant  comments  and  our  observations 

Overall  participants  found  the  Interactive  EFIT  prototype  easy  to  use.  When  commenting  on  the 
usability  of  the  interface  it  was  often  the  interactive  nature  that  participants  appreciated. 
Participants  appreciated  being  able  to  move  the  video  and  map  windows  around  and  position 
them  as  desired,  the  ability  to  pause  the  video  at  “key  features”  to  write  notes,  the  “user  friendly 
system  for  pulling  up  info  from  previous  IEDs  and  locations”  and  having  the  map  and  video 
synchronized. 

We  observed  that  all  participants  had  difficulty  switching  between  the  various  EFIT  tools.  Most 
participants  tried  to  open  up  the  IED  incident  reports  using  something  other  than  the  hyperlink 
tool.  Ideally  users  should  be  able  to  click  on  IEDs  using  the  selection  arrow  (that  is  used  to  play 
video  from  the  map).  Unfortunately  at  the  time  of  the  trial,  using  the  ArcMap  hyperlink  tool  was 
the  only  option  to  bring  up  the  incident  reports.  Users  should  also  be  able  to  pan  the  map  with  the 
selection  arrow,  avoiding  having  to  switch  to  the  pan  hand  tool.  An  onscreen  zoom  bar  (such  as 
Google  Maps™)  would  also  eliminate  the  need  to  switch  to  the  zoom  in  and  zoom  out  tools. 
Ideally  all  interaction  with  the  interface  could  be  done  with  one  tool,  the  selection  arrow. 

Participants’  interactions  with  the  map  highlighted  a  few  design  deficiencies: 

1.  The  direction  of  the  route  video  was  not  clear  on  the  KAF  to  Kandahar  City  and  MSG  to 
PBSG  videos.  The  yellow  progression  indicator  at  the  start  point  of  the  video  (see  Figure  9) 
was  not  intuitive.  When  the  video  was  not  playing  it  was  ambiguous  whether  the  blue  or 
yellow  indicated  the  forward  direction  of  travel.  Symbols  (triangles)  like  the  ones  used  for  the 
Kandahar  City  routes  (see  Figure  4)  should  be  used  to  indicate  the  video  direction  on  all  route 
video  tracks  or  video  start  points  should  be  distinctly  marked  from  the  end  points  (this  option 
is  less  helpful  when  the  start  and  end  point  are  outside  of  the  current  map  view).  The  direction 
indicators  should  have  consistent  size,  shape  and  colour  (those  used  in  the  prototype  were 
added  to  demonstrate  possible  options).  Participants  also  wanted  a  better  indication  of  where 
the  video  was  being  shot  from  on  the  map.  Participants  were  unclear  whether  the  end  of  the 
yellow  line  indicated  the  position  of  the  front  of  the  vehicle  or  the  location  of  the  camera 
mounted  on  the  vehicle.  Again  a  symbol  of  some  sort  (possibly  a  video  camera)  would  be 
useful  here.  An  optional  overlay  showing  the  camera’s  field  of  view  may  also  be  helpful. 

2.  As  participants  interacted  with  the  map  as  the  video  played,  they  would  lose  sight  of  the 
location  of  the  current  position  of  the  video  camera.  Participants  would  sometimes  be  zoomed 
too  far  into  the  map  and  the  yellow  indicator  would  traverse  off  the  map  or  they  would  want 
to  investigate  something  to  either  side  of  the  video  track  and  loose  the  camera  position.  For 


DRDC-RDDC-201 5-R009 


15 


novice  users  it  might  be  useful  to  have  the  map  always  centred  on  the  location  of  the  video 
camera.  For  more  comfortable  users  this  option  could  be  disabled  allowing  them  to  freely 
navigate  around  the  map.  A  “zoom  to  video  camera  position”  button,  like  the  “snap”  to  your 
current  GPS  location  feature  available  on  many  mobile  positioning/mapping  apps  (e.g., 
Google  Maps),  would  allow  advanced  users  to  quickly  reposition  the  map  on  the  video 
camera.  Additionally,  an  indicator  in  the  margin  of  the  map  window  indicating  the  direction 
of  the  current  location,  like  an  arrow  or  even  a  coloured  marker  that  forms  a  line  between  the 
centre  of  the  current  map  view  and  the  camera  location,  could  help  maintain  awareness  of  the 
position  of  the  inspected  area  relative  to  the  video  view,  which  might  make  some  people  more 
comfortable. 

3.  It  could  be  disorienting  for  participants  when  switching  from  one  route  video  to  another  using 
the  options  panel.  The  map  discretely  jumped  from  one  location  to  another.  This 
disorientation  could  be  prevented  with  a  smooth  transition  that  zooms  out  from  the  initial 
location,  transverses  the  map,  and  zooms  into  the  location  of  the  selected  video.  This 
technique  offers  “visual  momentum”  (Woods,  1984).  Labelling  tactical  features  such  as 
forward  operating  bases  would  also  improve  situation  awareness  (Endsley,  1995). 

4.  Map  symbols  representing  IED  incidents  should  not  be  the  same  colour  as  the  route  video 
tracks  as  they  may  be  appear  to  be  part  of  the  track. 

The  layout  of  the  video  alongside  the  map  as  shown  in  Figure  3  may  not  be  optimal.  Two 
participants  ignored  the  map  entirely  while  the  video  was  playing.  Another  participant  maximized 
the  map  and  placed  the  video  window  where  it  would  not  obscure  the  route  track  (Figure  9). 
Participants  who  considered  the  software  inflexible  may  have  appreciated  instructions  in  the 
EFIT  101  tutorial  on  how  to  resize  the  map  and  move  the  windows  around.  Participants  who 
found  the  software  inflexible  may  also  have  been  expressing  frustration  that  the  video  window 
could  not  be  resized. 
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Figure  9:  Example  of  the  map  maximized  behind  the  video  window.  Note  the  yellow  progress  bar 
in  the  bottom  right  corner.  As  the  video  track  is  played,  the  yellow  progress  indicator  extends 
along  the  blue  route  to  indicate  the  location  where  the  video  was  recorded. 

Some  participants  expected  more  context  for  the  Interactive  EFIT  videos.  Participants  wanted 
details  on  the  purpose  of  the  convoys  and  who  and  what  vehicles  were  in  them.  Sometimes 
participants  were  surprised  by  the  way  the  vehicle  crews  reacted  to  certain  situations,  but  felt  had 
they  had  more  information  about  the  convoy  they  might  be  able  to  understand  why  the  crew  acted 
as  they  did.  Participants  would  have  preferred  more  recent  video  footage  (<  6  months  old)  as 
insurgent  and  CAF  tactics  change  frequently. 

More  details  would  be  required  on  the  routes  listed  in  the  Options  Panel  if  the  number  of  routes 
within  Interactive  EFIT  were  to  grow.  Details  such  as:  start  location,  end  location,  route  name(s), 
date  of  video  recording  will  help  users  select  an  appropriate  video.  A  keyword  search  feature  may 
also  facilitate  efficient  video  selection. 

Other  observations  of  note:  Four  participants  ignored  the  options  panel  and  had  to  be  prompted  to 
use  it,  two  participants  did  not  use  or  recognize  map  IED  data,  two  participants  used  the  speaker 
volume  knob  to  increase  the  volume  of  the  video,  ignoring  the  software’s  volume  slider  and  one 
participant  often  pointed  and  touched  the  screen  to  indicate  locations.  One  participant  used  the 
map  to  perform  a  thorough  route  recce  before  watching  the  video  and  another  thought  selecting 
an  IED  incident  and  playing  the  video  would  give  a  playback  of  the  IED  strike. 

3.1.3  Feature  use 

Table  2  contains  a  summary  of  feature  use.  For  each  task  the  number  of  participants  who  used  the 
feature  is  indicated.  The  average  is  listed  in  the  rightmost  column.  Nearly  all  participants  used  the 
list  box  to  switch  between  videos  and  the  video  play  button  in  each  task.  Playing  a  video  from  the 
list  box  required  users  to  click  the  video  play  button  in  addition  to  selecting  the  video.  The 
number  of  participants  using  the  list  box  to  play  videos  drew  to  our  attention  that  having  users 
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select  a  video  is  sufficient  to  start  playing  the  video.  Future  prototypes  should  play  videos 
automatically  from  the  beginning  when  users  select  a  video  from  the  list  box. 


Table  2:  Frequency  of  participants  ’  Interactive  EFIT feature  use  by  task. 


Feature 

Task  1 

Task  2 

Task  3 

Task  4 

Average 

Pan 

6 

2 

4 

7 

68% 

Zoom 

6 

0 

3 

6 

54% 

Click  on  IED 

2 

2 

2 

5 

39% 

Change  audio  options  (radio  buttons) 

0 

4 

4 

4 

43% 

Switch  route  (list  box) 

6 

6 

5 

7 

86% 

Seek  on  video 

0 

2 

3 

5 

36% 

Seek  on  map 

0 

0 

1 

2 

11% 

Stop  (button) 

1 

1 

4 

4 

36% 

Play  video  (button) 

6 

5 

7 

7 

89% 

Change  volume 

2 

4 

5 

6 

61% 

Participants  did  not  switch  between  audio  options  for  the  first  task,  but  Tasks  2  through  4  were 
designed  with  the  expectation  that  participants  would  explore  this  option.  One  participant 
commented  that  “watching  the  video  first  with  live  Comms  and  then  again  with  dubbed  SME 
commentary  (is)  very  beneficial.” 

Only  three  of  the  participants  clicked  on  the  map  to  seek  through  a  video.  As  described  in 
Section  3.1.1  this  might  be  due  to  the  fact  that  the  yellow  progress  bar  was  not  described  when 
the  Interactive  EFIT  features  were  introduced.  However,  the  instructions  provided  did  describe 
clicking  on  the  route  video  tracks  to  play  a  video.  Since  this  feature  was  not  used  until  Task  3,  it 
is  likely  that  participants  were  unsure  how  this  feature  worked.  We  need  to  revisit  the 
intuitiveness  of  this  feature  and  how  to  best  train  users  to  associate  the  progress  of  the  video  with 
progress  on  the  map  (e.g.,  maybe  a  task  where  they  have  to  view  video  segments  specific  to  a 
geographic  location). 

We  also  asked  participants  how  many  times  they  would  want  to  watch  the  video  in  Task  1  before 
they  felt  confident  with  the  route.  Participants  expected  they  would  view  the  video  two  to  twelve 
times.  Participants  felt  they  would  watch  the  video  a  few  times  with  pauses  and  a  few  times 
watching  it  uninterrupted. 

3.1.4  Software  bugs 

As  participants  used  the  Interactive  EFIT  prototype  we  noticed  a  few  software  bugs  that  were  not 
observed  during  preparation: 

1 .  The  map  data  disappeared  sporadically  after  participants  panned  and  zoomed. 

2.  Since  the  EFIT  arrow  is  the  ArcMap  select  features  tool,  it  allowed  participants  to  click  and 
drag  on  the  map  to  select  multiple  segments  of  a  video  route  track.  Participants  who  clicked 
and  dragged  over  a  video  track  for  one  reason  or  another  were  not  expecting  to  be  able  to 
select  segments  of  the  track  (the  track  appears  continuous).  Once  the  track  segments  were 
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selected,  participants  did  not  know  how  to  unselect  them  and  resume  their  previous  activity. 
Video  track  route  segments  should  not  be  selectable. 

3.  Similarly  to  bug  #2,  when  participants  clicked  on  features  other  than  the  video  track  using  the 
EFIT  arrow,  the  features  would  highlight.  The  software  did  not  tell  participants  what  the 
highlight  meant  and  did  not  provide  them  with  additional  actions  when  highlighted.  Map 
features  should  not  highlight  when  clicked  in  future  prototypes  unless  users  can  interact  with 
them  in  some  way. 

4.  The  IED  database  received  from  the  CAF  Mapping  and  Charting  Establishment  appears  to 
have  some  errors  in  the  location  of  incidents.  Some  IEDs  are  placed  in  inexplicable  locations. 
Errors  may  be  due  to  the  precision  of  the  grid  reference  used  when  reporting. 

5.  We  noticed  that  when  a  playing  video  reached  the  end  and  stopped  the  map  would  refresh 
many  times.  On  occasion  the  map  entered  a  cycle  where  it  infinitely  refreshes,  crashing  the 
software. 

6.  Occasionally  the  yellow  progress  indicator  for  the  video  routes  on  the  map  did  not  update 
correctly  to  show  the  correct  position  of  the  video  camera. 

7.  Clicking  on  IED  incidents  with  the  lightning  bolt  hyperlink  tool  to  open  incident  reports 
requires  a  high  degree  of  placement  precision.  A  wider  margin  of  error  for  cursor  placement 
should  be  implemented  in  future  prototypes. 

These  bugs  need  to  be  resolved  before  further  user  testing.  We  recommend  that  tooltip  hover 
boxes  be  added  to  the  EFIT  tools.  Tooltips  provide  information  in  a  hover  box  when  users  hover 
the  mouse  pointer  over  an  item  without  clicking  it.  Tooltips  can  also  be  used  with  Interactive 
EFIT  to  show  more  information  when  users  hover  over  video  tracks  and  IED  incident  data. 

3.2  EFIT’s  training  potential 

3.2.1  Usefulness  questionnaire 

The  seven  participants  were  asked  to  rate  the  usefulness  of  Interactive  EFIT  for  seven  learning 
objectives.  Average  results  for  the  participants  in  this  study  are  shown  in  Table  3  alongside  the 
32  students  and  11  instructors  from  2CER  TMST  TF  3-08  ETHAR  and  the  49  students  from 
TF  3-09  1CER  training  lectures  who  watched  the  two  non-interactive  video  prototypes  described 
in  Section  1.2.  The  top  three  ratings  for  each  group  are  highlighted  in  green.  The  TF  3-08, 
TF  3-09  and  Interactive  EFIT  data  is  contrasted  here  to  get  a  sense  of  the  relative  strengths  and 
weaknesses  of  the  different  prototypes. 
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Table  3:  Average  usefulness  ratings  of  EFIT for  learning  specific  skills.  The  top  three  scores  for 

each  group  are  highlighted  in  green. 


Usefulness  of  EFIT  for  learning  the 
following  (1  =  not  at  all  useful;  5  = 
extremely  useful) 

TF  3-08 
Non- 

Interactive 

EFIT 

students 

(n=32) 

TF  3-08 
Non- 

Interactive 

EFIT 

instructors 

(n=ll) 

TF  3-09 
Non- 

Interactive 

EFIT 

students 

(n=49) 

Interactive 

EFIT 

(n=7) 

1)  Route  familiarization 

3.4 

3.6 

3.6 

4.7 

2)  Theatre-specific  terrain 
familiarization 

4.4 

4.3 

3.9 

4.6 

3)  Theatre-specific  cultural  environment 
familiarization  (“what  is  normal”) 

4.1 

3.9 

4.0 

3.4 

4)  Detection  of  IED  indicators 

4.3 

4.3 

4.1 

4.3 

5)  Assessment  of  IED  threat 

4.0 

4.1 

4.0 

4.0 

6)  Introducing  context  of  convoy 
experience 

3.8 

4.0 

3.8 

3.4 

7)  Training  for  scanning  an  arc 

3.2 

3.6 

3.5 

2.6 

Average 

3.9 

4.0 

3.8 

3.9 

We  note  here  that  a  statistical  comparison  of  the  usefulness  ratings,  both  within  each  user  group, 
and  between  the  groups,  is  problematic  for  a  number  of  reasons.  The  number  of  participants  in 
each  group  is  very  uneven  (32,  11,  49  and  7  for  the  TF3-08  students,  TF3-08  instructors,  TF3-09 
members  and  the  PSTC  participants,  respectively),  thus  making  it  very  unlikely  that  the  combined 
data  set  would  meet  the  variance  assumptions  required  for  inferential  statistics.  Perhaps  more 
importantly,  as  these  data  were  not  collected  under  a  systematic  experimental  design,  many 
uncontrolled  factors  varied  between  each  group,  in  addition  to  the  EFIT  design  itself,  making 
between-group  differences  very  difficult  to  interpret.  Furthermore,  the  different  utility  ratings  are 
not  systematically  related  to  each  other,  and  merely  represent  users’  subjective  impressions 
regarding  the  utility  of  different  features  of  EFIT  preferences.  Thus,  it  is  again  difficult  to 
interpret  a  significant  difference  between  the  ratings  of  different  EFIT  features.  Nevertheless, 
there  is  some  value  in  knowing  whether  the  response  patterns  of  the  different  groups  actually 
differed.  For  this  reason,  the  ratings  presented  in  Table  3  were  subjected  to  a  mixed  design, 
between-within  analysis  of  variance,  with  group  (TF3-08,  TF3-09  and  PSTC)  as  the  between- 
subjects  factor,  and  the  7  usefulness  ratings  as  the  within-subjects  factor,  with  a  significance  level 
of  a  =  .05.  (The  two  subgroups  of  TF3-08  were  pooled  together,  as  a  preliminary  analysis  showed 
no  statistical  differences  between  these  two  subgroups).  The  analysis  revealed  a  main  effect  of 
ratings  (F(6,558)  =  13.1,  MSE  =  0.54)  and  an  interaction  between  group  and  ratings 
(F(12,558)  =  3.8,  MSE  =  0.54).  This  suggests  that  the  different  groups  rated  the  7  EFIT  features 
they  were  asked  about  differently,  and  that  the  pattern  of  preferences  varied  between  groups. 
However,  for  the  reasons  given  above,  no  further  analyses  were  performed,  and  participants’ 
ratings  are  discussed  hereafter  without  reference  to  their  statistical  significance. 

Participants  using  Interactive  EFIT  found  it  most  useful  for  route  familiarization.  The  interactive 
prototype  scored  higher  than  the  non-interactive  videos  for  route  familiarization.  As  noted  above, 
the  reason  for  this  preference  is  difficult  to  ascertain.  We  feel  that  that  panning  and  zooming  on 
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the  map  likely  improved  perceptions  of  Interactive  EFIT’s  usefulness  for  route  familiarization. 
However,  the  difference  in  preferences  could  also  be  due  to  other  differences  between  groups,  for 
instance,  different  operational  experiences,  or  even  simply  the  fact  that  so  few  participants  used 
Interactive  EFIT  relative  to  the  non-interactive  version.  Determining  the  cause  of  these 
differences  is  beyond  the  scope  of  this  study  and  would  require  further  work. 

Second  to  route  familiarization,  participants  felt  the  interactive  prototype  was  also  useful  for 
theatre-specific  terrain  familiarization.  This  high  rating  is  consistent  with  the  non-interactive 
videos.  Also  consistent  with  the  non-interactive  videos  was  the  perceived  usefulness  of 
Interactive  EFIT  to  teach  the  detection  of  IED  combat  indicators.  The  ratings  show  EFIT  has 
good  potential  as  a  training  tool  and  warrants  further  study.  Participants  felt  the  Interactive  EFIT 
prototype  was  least  useful  for  training  arc  scanning. 

Participants  were  also  asked  to  rate  the  usefulness  of  adding  features  to  EFIT.  Results  for  the 
seven  PSTC  SMEs  who  used  Interactive  EFIT  are  shown  in  Table  4  alongside  the  32  students  and 
11  instructors  from  2CER  TMST  TF  3-08  ETHAR  and  the  49  students  from  TF  3-09  1CER 
training  lectures  who  watched  the  two  non-interactive  video  prototypes  described  in  Section  1.2. 
The  top  three  ratings  for  each  group  are  highlighted  in  green.  Participants  were  not  asked  to  rate 
features  rated  N/A.  For  the  most  part  these  features  had  already  been  implemented  in  the 
Interactive  EFIT  prototype.  (As  noted  above,  no  statistical  analysis  of  the  responses  was 
performed  here,  as  statistical  differences  between  ratings  for  hypothetical  EFIT  features  are  even 
more  difficult  to  interpret  than  between  ratings  for  actual  features). 


Table  4:  Average  usefulness  ratings  for  EFIT  additions  and  improvements.  The  top  three  scores 

for  each  group  are  highlighted  in  green. 


Additions  and  Improvements  to 
EFIT  (1  =  not  at  all  useful;  5  = 
extremely  useful) 

TF  3-08 
Non¬ 
interactive 
EFIT 
students 
(n=32) 

TF  3-08 
Non¬ 
interactive 
EFIT 

instructors 

(n=ll) 

TF  3-09 
Non¬ 
interactive 
EFIT 
students 
(n=49) 

Interactive 

EFIT 

(n=7) 

1)  Incorporate  videos  of  the  same 
route  taken  at  different  times  of  day 

4.1 

4.4 

3.9 

4.1 

2)  Improve  visibility  of  indicator  cues 

3.6 

3.9 

3.4 

N/A 

3)  Add  image  intensified  (II)  video 

3.3 

3.8 

3.5 

4.5 

4)  Add  infrared  (IR)  video 

3.3 

3.5 

3.5 

4.6 

5)  Add  locations  of  past  IED  incidents 
on  route  map 

4.3 

4.1 

4.2 

N/A 

6)  Add  details  of  past  IED  incidents 
on  route  map 

4.0 

3.2 

4.1 

4.6 

7)  Add  locations  of  OPFOR  activity 
on  route  map 

3.7 

3.1 

3.9 

4.6 

8)  Highlight  locations  of  canalizing 
ground  on  route  map 

3.3 

3.6 

3.7 

4.0 

9)  Remove  map 

3.2 

2.5 

2.2 

1.0 

10)  Add  locations  of  pogo-points  / 
waypoints  /  checkpoints  /  destinations 
on  route  map 

3.0 

3.5 

3.0 

3.9 
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Additions  and  Improvements  to 
EFIT  (1  =  not  at  all  useful;  5  = 
extremely  useful) 

TF  3-08 
Non¬ 
interactive 
EFIT 
students 
(n=32) 

TF  3-08 
Non¬ 
interactive 
EFIT 

instructors 

(n=ll) 

TF  3-09 
Non¬ 
interactive 
EFIT 
students 
(n=49) 

Interactive 

EFIT 

(n=7) 

11)  Add  the  option  to  switch  between 
a  map  mode  vs.  the  satellite  imagery 

3.2 

2.8 

3.0 

3.3 

12)  Add  zoom  functionality  to  the 
map 

3.6 

2.8 

3.2 

N/A 

13)  Show  alternate  routes  to  destinations 
on  route  map 

3.8 

3.3 

3.5 

3.6 

14)  Indicate  the  date  and  time  of  day 
the  video  was  taken 

3.3 

2.2 

3.0 

3.9 

15)  Add  the  name  of  the  route  the 
video  was  taken 

3.1 

2.2 

2.8 

4.3 

16)  Make  the  map  interactive  so  that 
routes  can  be  clicked  on  to  play  video 
sequence 

3.8 

3.6 

3.4 

4.3 

17)  Add  simulation  demonstrations  of 
TTPs 

4.0 

4.4 

3.7 

N/A 

1 8)  Increase  size/length  of  video 

4.0 

3.6 

3.4 

3.6 

19)  Different  camera  views  (2nd 
vehicle  in  OOM,  rear  view,  gun  turret 
view) 

4.3 

4.3 

3.9 

4.4 

Participants  who  used  Interactive  EFIT  felt  the  most  useful  additions  were  adding  image 
intensified  video,  details  of  past  IED  incidents  on  the  route  map,  and  adding  locations  of  opposing 
force  (OPFOR)  activity  on  route  map.  Having  interacted  with  the  map  for  the  four  testing  tasks, 
users  of  Interactive  EFIT  saw  no  usefulness  in  removing  the  map.  Participants  using  Interactive 
EFIT  also  rated  highly  adding  infrared  video  and  different  camera  views  (e.g.,  2nd  vehicle  in 
OOM,  rear  view,  gun  turret  view). 

In  addition  to  the  features  listed  in  the  questionnaire  results  (Table  4)  above,  the  seven  PSTC 
participants  suggested  additional  features  that  they  would  like  to  see  in  future  prototypes. 
Participants  wanted  more  information  on  the  buildings  alongside  the  route.  They  requested 
building  heights  and  strategic  importance,  if  any.  Participants  wanted  video  with  a  larger  field  of 
view  and  videos  from  the  various  vehicle  platforms  (Leopard  2,  Light  Armoured  Vehicle  III, 
Tracked  Light  Armoured  Vehicle,  etc.).  One  participant  suggested  that  the  instructional  SME 
video  should  pause  occasionally  throughout  playback  to  highlight  and  describe  important 
features. 

Participants  also  wanted  to  be  able  to  mark-up  the  map  with  possible  IED  locations  and  terrain 
features.  Participants  envisioned  a  tool  with  these  features  would  be  useful  in  theatre  prior  to 
departing  on  a  mission  outside  of  the  wire:  “(I  would)  look  at  (the)  system  the  night  before  to  plan 
and  get  a  sense  of  what’s  there.” 
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When  asked  if  they  learned  anything  new  while  using  Interactive  EFIT  that  was  not  included  in 
previous  training,  two  participants  did  not  answer  the  question,  one  participant  said  “no”  and  the 
other  four  participants  wrote: 

1 .  “This  provided  excellent  famil(iarization)  with  terrain,  pattern  of  life  and  road  conditions. 
Great  acclimatization.” 

2.  The  “actual  view  of  roads  I  would  use.  With  time  (I)  could  clearly  identify  possible  IED  areas 
and  choke  points.  (Interactive  EFIT)  could  definitely  help  plan  for  convoy  ops  in  the  future.” 

3.  “Video  of  ground,  reactions  of  vehicles,  reactions  of  people.  Actual  possible  IED  encounter 
that  can't  be  portrayed  in  pictures.” 

4.  “This  would  be  a  ‘first’  for  seeing  footage  of  the  ground  before  being  in  theatre  for  the  first 
time.” 

All  participants  indicated  Interactive  EFIT  has  potential  training  benefits.  Participants  felt 
Interactive  EFIT  could  be  used  for  the  following  teaching  points: 

1.  “With  more  videos  of  different  times  you  can  learn  local  traffic  patterns  thus  allowing  you  to 
plan  better  convoys.” 

2.  “Something  that  is  impossible  to  replicate  is  seeing  Kandahar  for  yourself,  (Interactive  EFIT) 
allows  (students)  to  do  that.  Any  of  the  crowded  villages  also,  you  wouldn’t  believe  the  crazy 
things  happening  there,  but  to  see  it  beforehand  (is  useful).” 

3.  You  can  “realize  the  terrain  you  will  be  in  -  cars  all  over  the  road,  choke  points,  etc.  (They) 
are  a  concern  but  there’s  nothing  really  you  can  do  about  it.” 

4.  You  can  recognise  “built  up  areas  (on  map)  which  are  possible  ambush  locations. .  .culverts. . . 
routes  into  and  out  of  the  route  we  are  travelling,  which  could  be  used  as  VBIED  routes.” 

5.  “Firsthand  view  of  interaction  between  convoy  and  locals.” 

6.  Learn  routes  before  entering  theatre  and  review  routes  for  changes  since  last  tour. 

7.  Communications. 

3.2.2  Verbal  comments  on  EFIT  uses 

Prior  to  the  trial  we  anticipated  Interactive  EFIT  would  be  used  by  ETHAR  trainers  for  briefing 
IED  threats  during  TMST.  We  thought  the  main  benefit  over  traditional  training  technologies 
would  be  the  visualization  of  IED  threats  within  the  environment  in  which  they  are  likely  to 
occur.  We  also  anticipated  that  troops  could  benefit  from  using  Interactive  EFIT  on  their  own  to 
explore  the  route  within  their  AO  and  become  familiar  with  local  threats.  Trial  participants  also 
identified  these  uses  and  offered  other  ways  Interactive  EFIT  could  add  value  to  the  CAF: 
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1.  Many  participants  suggested  Interactive  EFIT  could  be  watched  in  vehicle  crew  groups. 
Crews  would  be  instructed  to  “critic  [sic]  flaws  and  point  out  things  done  well,  practice 
Comms,  call  out  threats  and  reactions  to  indicators.”  Participants  felt  crew  members  would 
learn  from  each  other,  “bring(ing)  up  points  that  others  will  have  missed.”  An  SME  or 
instructor  could  provide  feedback  on  how  the  group  did. 

2.  Before  navigating  a  route,  crew  commanders  could  be  brought  together  to  watch  Interactive 
EFIT  videos  together.  Crew  commanders  would  “discuss  how  the  route  will  be  navigated: 
‘when  we  get  here  we  will  slow  down,  here  we  will  speed  up  if  we  can,  (and)  obviously 
terrain  and  traffic  will  still  dictate’.  Crew  commanders  could  also  discuss  “past  IED  sites  and 
how  the  area  will  be  dealt  with.” 

3.  Integrate  Interactive  EFIT  into  Rules  of  Engagement  training. 

4.  Individual  learning  and  training  (i.e.,  “Language  Lab”). 

5.  Gunners  and  crew  commanders  could  use  Interactive  EFIT  together  to  practice  communications 
(threat  identification). 

6.  Drivers  could  use  Interactive  EFIT  to  simulate  route  drives.  We  expect  360  degree  video  and 
a  physical/motor  component  may  be  required. 

7.  Route  familiarization  with  paper  map  that  will  be  used  in  theatre:  “Watch  the  video,  track 
progress  on  computer  map,  locate  current,  start  and  end  positions  on  paper  map,  then  phase 
out  the  computer  map  so  they  can  track  progress  and  find  their  location  using  the  paper  map 
(since  they  won’t  have  the  computer  map  in  the  vehicle  with  them  but  they  could  easily  have 
a  paper  map  in  the  vehicle).” 

3.3  Trial  procedure  observations 

As  documented  above,  this  Interactive  EFIT  user  trial  provided  useful  feedback  for  future 
iterations  of  development.  We  recommend  that  as  new  prototypes  are  developed  further  user 
testing  is  conducted.  This  section  outlines  a  few  lessons  learned  from  the  conduct  of  the  trial  that 
we  would  consider  doing  differently  in  future  trials. 

As  noted  in  Section  3.1.1  we  did  not  provide  participants  with  instructions  on  how  the  colour  of 
the  video  track  changes  (from  blue  to  yellow)  when  playing  a  video  to  indicate  the  position  of  the 
video  camera.  Our  misstep  not  documenting  this  feature  provided  additional  unexpected  feedback 
on  the  interface  feature.  The  think  aloud  technique  provided  us  with  feedback  on  what 
participants  thought  the  change  in  colour  meant,  thus  providing  a  measure  for  the  intuitiveness  of 
the  interface  feature.  In  future  studies  where  measures  of  the  intuitiveness  of  an  interface  are 
sought  we  recommend  intentionally  not  documenting  features  and  asking  participants  to  explain 
the  function  of  the  interface  feature. 

There  were  a  few  interactions  with  the  interface  that  we  did  not  record.  During  the  four  tasks 
participants  moved  the  video  and  map  windows  much  more  than  we  had  expected.  Future  studies 
should  record  the  number  of  times  windows  are  relocated,  resized  and  their  positions  on  the 
desktop.  Feedback  on  user  preferences  for  windows  locations  and  size  can  be  used  to  organize  the 
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windows  in  an  optimal  arrangement  when  users  first  begin  using  the  software.  If  different 
arrangements  are  preferred  for  different  tasks,  we  could  facilitate  the  rearrangement  of  windows 
through  interface  design  techniques,  such  as  an  interface  that  learns  the  user’s  preferences  and 
adapts  appropriately  for  the  task. 

We  also  did  not  record  the  use  of  window  scroll  bars  or  clicks  on  features  that  we  explicitly  asked 
participants  not  to  use.  Three  participants  used  their  mouse  cursor  to  scan  the  terrain  of  the 
playing  video.  It  would  have  been  useful  to  quantify  this  behaviour.  We  had  intended  to  use 
Morae  (TechSmith,  Okemos,  Michigan)  software  to  record  user  interactions  with  the  prototype, 
but  due  to  time  constraints  we  were  not  able  to.  We  recommend  using  a  tool  like  Morae  which 
records  detailed  user  interactions  with  computer  applications  for  future  studies. 

As  mentioned  in  Section  3.1.1  noise  from  an  HVAC  system  made  it  difficult  for  participants  to 
hear  the  audio  tracks  of  the  Interactive  EFIT  videos.  The  HVAC  noise  also  reduced  the  audio 
quality  of  the  session  video  recordings.  Some  participant  statements  on  the  video  were  not 
intelligible.  We  recommend  insisting  on  a  quiet  room  when  making  arrangements  for  on-site  user 
testing. 

When  deploying  questionnaires  in  future  user  testing,  we  should  discuss  responses  with  the 
participants.  Discussing  high  and  low  ratings  will  reduce  post  hoc  interpretations  of  why 
participants  rated  items  the  way  they  did.  Future  trials  with  quantitative  performance  measures 
(speed,  accuracy  on  completion  of  specific  task  goals,  correctness  of  procedures  followed,  etc.) 
and  quantitative  evaluations  of  training  effectiveness  would  also  be  helpful  to  validate  (or  not) 
participant  opinions. 
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4  Conclusions 


Overall  participants  found  Interactive  EFIT  to  be  an  easy  to  use  and  felt  it  has  potential  as  a 
training  tool  for  the  detection  of  IED  combat  indicators  and  theatre-specific  route  and  terrain 
familiarization. 

That  said,  feedback  from  the  usability  questionnaire  suggests  the  software  needs  to  be  made  more 
flexible.  Allowing  users  to  resize  the  video  window  would  be  a  great  first  step  to  improve 
flexibility.  Participants  also  did  not  find  Interactive  EFIT  overly  stimulating. 

We  need  to  reconsider  the  intuitiveness  of  the  yellow  video  progress  bar  on  the  map,  vehicle  crew 
Comms  should  not  be  simulated  where  actual  Comms  do  not  exist  and  the  number  of  tools 
required  to  interact  with  the  map  should  be  reduced. 

We  need  to  fix  documented  software  bugs.  Most  urgently  we  need  to  address  bugs  that  prevent 
the  map  from  updating  correctly  or  cause  the  software  to  crash. 

Participants  identified  image  intensified  video,  details  of  past  IED  incidents  on  the  route  map,  and 
adding  locations  of  OPFOR  activity  on  the  map  as  priorities  for  new  features  to  be  implemented. 

Despite  the  changes  that  need  to  be  made  participants  found  Interactive  EFIT  suitable  to  teach: 
local  traffic  patterns,  village  behaviour,  ambush  locations,  culverts,  alternate  routes,  possible  IED 
areas,  and  communications.  They  also  thought  Interactive  EFIT  would  be  useful  for:  crew 
discussion  sessions,  in-theatre  crew  commander  planning  meetings,  rules  of  engagement  training, 
individual  terrain  familiarization,  communication  of  threat  identification,  and  driving  simulations. 

Comments  from  participants  highlight  multiple  (possibly  competing)  uses  for  Interactive  EFIT  as 
a  Counter-IED  training  tool  and  an  intelligence  tool  (either  for  training  or  planning). 

We  recommend  implementing  the  feedback  from  this  evaluation  into  future  Interactive  EFIT 
prototypes.  Further  user  evaluations  should  also  be  conducted  with  a  usability  testing  tool  such  as 
Morae  (TechSmith,  Okemos,  Michigan). 

A  separate  study  was  conducted  to  evaluate  the  training  effectiveness  of  EFIT  video  content  (not 
of  the  Interactive  EFIT  features  evaluated  here).  That  study  found  that  EFIT  can  be  effective  in 
improving  soldiers’  IED  threat  judgements  (Jarmasz  et  al.,  2010).  Together  the  usability  study 
reported  here  and  Jarmasz’ s  effectiveness  study  suggest  that  EFIT  can  be  a  training  tool  with 
good  effectiveness  and  usability,  and  highlight  improvements  that  could  make  it  more  so. 
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Annex  A  Early  EFIT  prototypes 


Figure  A.l:  Screenshot  of  the  first  EFIT  prototype  with  ground  vehicle  video  of  Afghanistan 
beside  aerial  imagery  showing  the  moving  position  of  the  camera  (yellow  line). 


Figure  A.2:  Screenshot  of  the  second  EFIT  prototype. 
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Figure  A.3:  Paper  prototype  of  an  interactive  version  ofEFIT. 
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Annex  B  Video  commentary 


B.1  KAF  to  Kandahar  City 

Filmed  in  Kandahar  May  2007. 

VIDEO  COMMENTARY 

You  can  see  a  large  advertising  sign  off  to  the  left  which  would  make  a  very  good  aiming  marker. 
The  jingle  truck  ahead  pulls  off  the  road  but  remains  close  to  the  convoy  and  could  hide  a  large 
amount  of  explosives.  There  is  no  civilian  infrastructure  or  people  in  the  area  which  could  also 
indicate  an  attack.  However  the  ground  is  wide  open  to  the  right. 

There  are  a  number  of  culverts  on  this  road  that  could  conceal  IEDs.  The  trees  or  telephone  poles 
on  the  right  are  potential  aiming  markers  and  the  buildings  to  the  left  provide  observation 
positions  and  could  conceal  firing  points.  However,  the  ground  here  again  is  wide  open. 

Coming  up  on  the  right  are  a  number  of  fuel  trucks  very  close  to  the  edge  of  the  road.  There  is 
also  a  lack  of  local  personnel  in  the  area.  However  with  a  detonation  here  there  would  be  a  lot  of 
collateral  damage  to  the  infrastructure. 

The  truck  seems  to  have  the  convoy  canalized  between  him  and  the  large  pile  of  rock  on  the  left, 
either  of  which  could  have  an  IED  concealed  in  them. 

Here  the  IED  could  be  in  the  culvert  under  the  road  with  the  end  of  the  wall  acting  as  an  aiming 
marker. 


Table  B.l:  SME  notes  used  for  video  commentary  (KAF  to  Kandahar  City). 


Time 

Cue 

Positive  IED  Indicator(s) 

Negative  IED 
Indicator(s) 

0:08 

Advertising  sign  and 
large  truck  on  the  left 

The  sign  is  a  possible  aiming 
marker,  truck  is  very  close,  could 
be  a  SVBIED,  no  civilian 
infrastructure  or  personnel  in  the 
area 

The  ground  is  wide  open 
so  the  convoy  could  veer 
to  the  right  to  avoid  the 
truck 

0:08- 

0:42 

Culverts,  trees, 
telephones  poles, 
buildings  in  general 

There  are  a  number  of  culverts 
on  this  road  which  could  conceal 
IEDs,  the  trees  or  building  to  the 
left  could  be  concealing  a  firing 
point  and  the  telephone  poles  and 
trees  to  the  right  might  be  used 
as  aiming  markers 

The  ground  is  wide  open 

0:42 

Fuel  trucks  on  the 
right 

Trucks  are  very  close  to  the  road, 
lack  of  people  in  the  area 

There  would  be  a  lot  of 
collateral  damage  to  the 
infrastructure 
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Time 

Cue 

Positive  IED  Indicator(s) 

Negative  IED 
Indicator(s) 

0:57 

Debris  on  the  left, 
truck  on  the  right 

Truck  seems  to  have  the  convoy 
canalized  between  him  and  the 
debris/rocks  to  the  left,  either  of 
which  could  have  IEDs 
concealed  in  them 

1:05 

Culvert  with  built  up 
area  on  the  left 

Culvert  could  hide  an  IED  and 
the  end  of  the  wall  could  be  used 
as  an  aiming  marker,  firing  point 
could  be  in  the  built  up  area  to 
the  left 

1:21 

White  car  from  the 
left 

White  car  approaches  from  the 
left  and  crosses  the  road  ahead  of 
the  convoy 

Clears  the  road  before  the 
convoy  passes 

B.2  Downtown  Kandahar  City 

Filmed  in  Kandahar  Dec  2005. 

VIDEO  COMMENTARY 

You  can  see  a  concrete  medium  off  to  the  left.  It  protects  the  convoy  from  oncoming  traffic  or 
VBIEDs  from  the  left  but  at  the  same  time  it  could  be  used  to  conceal  explosives  in  the  concrete 
itself,  and  if  there  is  an  ambush  or  IED  attack  it  will  limit  the  convoy’s  movement. 

You  have  a  parked  car  in  the  middle  of  the  road,  directly  ahead,  between  it  and  the  concrete 
medium  you  have  a  choke  point,  therefore  the  car  is  a  potential  VBIED,  but  the  local  police  seem 
to  be  dealing  with  it  and  the  locals  aren’t  avoiding  it. 

You  see  a  stone  arch  to  the  front;  it’s  fairly  narrow,  leading  to  a  traffic  circle.  The  arch  could 
conceal  a  large  amount  of  explosives  but  may  not  be  a  good  target  depending  on  the  religious  or 
cultural  significance  of  the  arch  itself.  There’s  a  drop  down  gate  so  the  convoy  has  to  slow  down 
or  come  to  a  stop  before  entering  the  round-about  which  in  itself  can  be  quite  chaotic  with  all  the 
pedestrians  and  traffic.  This  is  a  good  example  of  a  choke  point  as  there  is  no  other  by-pass. 

You  will  see  a  yellow  car  off  to  the  right,  the  2nd  one,  which  veers  the  convoy  to  the  left.  The 
black  truck  on  the  left  could  be  a  good  spot  for  an  IED  with  the  telephone  pole  acting  as  an 
aiming  marker.  But  the  street  is  very  busy  with  people  and  the  truck  is  too  far  off  the  road  to 
cause  significant  damage  to  armoured  vehicles  if  detonated. 

Same  situation  again,  the  yellow  taxi  channels  the  convoy  to  the  left  where  the  black  SUV  is 
pointed  at  the  convoy,  possibly  a  suicide  VBIED.  There  are  many  possible  firing  points  from 
within  the  buildings  on  the  right.  Depending  on  your  ROEs  warning  shots  may  have  been 
required  in  both  those  incidents. 
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We’re  coming  up  on  a  piece  of  carpet  on  the  road  to  your  right  which  could  camouflage  a 
pressure  plate  IED.  Possibly  not  much  of  a  threat  because  so  many  cars  have  driven  over  it  and  it 
is  a  paved  road  with  lots  of  people  about. 


Table  B.2:  SME  notes  used  for  video  commentary  (Downtown  Kandahar  City). 


Time 

Cue 

Positive  IED  Indicator(s) 

Negative  IED  Indicator (s) 

0:00 

Concrete  medium  on 
the  left  (close 
proximity  to  convoy) 

Could  conceal  explosives 
and/or  limit  convoy’s 
movement  in  case  of  an 
attack/ambush 

Protects  convoy  from  oncoming 
traffic  (VBIED)  from  the  left 

0:15 

Stopped/parked  car  on 
road  directly  in  front 
of  convoy 

Potential  VBIED  waiting 
for  convoy  to  pass  in  close 
proximity  (choke  point) 

ANP  on  site  dealing  with  it, 
vehicle  appears  new/good 
condition,  locals  not  avoiding  it 

0:50 

Large  stone  arch  at 
end  of  street  leading  to 
traffic  circle 

Arch  could  conceal  a  large 
amount  of  explosives, 
narrow  passage  with  gate 
which  means  convoy  has  to 
slow  down/stop  (choke 
point) 

Arch  may  not  be  a  good  target 
depending  on  the 
religious/cultural  significance, 
ANP  and  many  local  people 
around 

1:22 

Choke  point  between 
tuk-tuk/rick-shaw/ 
wagon  and  yellow  car, 
note  the  black  truck 
beside  telephone  pole 
on  left  immediately 
after 

Yellow  car  on  right,  veers 
convoy  to  the  left,  either  the 
wagon  or  the  black  truck  on 
left  could  be  a  VBIED, 
telephone  pole  by  the  truck 
is  a  possible  aiming  marker, 
buildings  on  the  right  allow 
for  good  overview  (possible 
firing  point). 

(Warning  shot  required?) 

Street  is  still  very  active  with 
public,  black  truck  is  too  far  off 
to  the  left  (off  the  road)  to  cause 
significant  damage  to  convoy  if 
detonated 

1:36 

Choke  point  between 
yellow  taxi  on  right 
and  2  white  minivans 
on  the  left,  black  SUV 
on  left  pointed  at  the 
convoy 

Yellow  taxi  on  right  cuts 
the  convoy  off,  black  SUV 
on  the  left  is  directed  at  the 
convoy,  possible  SVBIED, 
very  close  proximity 
(Warning  shot  required?) 

Too  many  locals/people  around 

1:54 

Carpet  on  road  (right 
side) 

Possible  camouflage  for 
pressure  plate  IED 

Lots  of  traffic  driving  over  it, 
the  road  is  paved,  lots  of  people 
around 

B.3  MSG  to  PBSG 

Filmed  in  Kandahar  November  2007. 
VIDEO  COMMENTARY 
( )  -  contains  descriptive  text 
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Table  B.3:  Video  commentary  (MSG  to  PBSG). 


Time 

Person  Talking 

Commands/Comms 

0m21s 

Crew  Comd 

Eric,  reference  100  meters  up  there,  there’s  a  big  gap  in  the  wall 
on  the  left  hand  side,  just  keep  your  eyes  on  it 

0m27s 

Gunner 

Roger,  I  got  it 

Om39s 

Crew  Comd 

Oh,  here  comes  buddy  on  a  bike.  All  c/s  1 1  be  advised  lone  male 
passing  on  the  left  side  of  the  vehicles  now,  out. 

lm51s 

Crew  Comd 

Matt,  let’s  not  drive  like  we’re  on  the  country  side,  let’s  pick  up 
the  speed  a  bit. 

lm58s 

Driver 

I’m  trying  to  do  the  best  I  can 

2m02s 

Crew  Comd 

That’s  what  you’re  always  telling  us,  let’s  just  go 

2m26s 

Crew  Comd 

Alright  we’re  approaching  the  ANA. .  .correction,  ANP 
compound  off  to  the  right  hand  side  there 

2m48s 

Crew  Comd 

Alright  well  the  ANA  Chicane  is  coming  up  there,  so  let’s  slow 
down  over  here 

2m55s 

Roger 

3m22s 

Crew  Comd 

All  c/s  11,  this  is  1 1A,  approaching  another  village,  out 

3  m3  Os 

PI  Comd  12 

1  this  is  12,  message,  over 

3m33s 

PI  Comd  1 

1,  send 

3m35s 

PI  Comd  12 

12  roger,  conducting  a  short  halt  with  ANP  PERS  at  grid 
QQ3012589627.  We  will  be  moving  shortly  back  to  MSG.  We 
will  be  there  in  20  Mikes,  over 

3m50s 

PI  Comd  1 

1,  roger. 

3m54s 

Crew  Comd 

Heyhey,  Matt  take  a  look,  there’s  those  kids  on  the  side  of  the 
road  again,  the  ones  that  threw  the  rocks  at  us.  Wonder  if  they’re 
gonna  throw’em.  Yep,  there  he  goes.  Alright  guys,  be  aware  those 
kids  are  throwing  rocks  at  us  again. 

4m08s 

Driver 

Want  me  to  swerve  at  one? 

4m09s 

Crew  Comd 

(snickering)  No,  no,  no.  I  want  to  try  to  get  out  without  running 
over  a  kid,  I  don’t  know  if  it’s  possible  for  sure 

5m31s 

Crew  Comd 

All  c/s  1 1  this  is  1 1  A,  be  advised  that  we’re  exiting  the  village 
and  approaching  more  open  terrain,  out 

6ml9s 

Crew  Comd 

Matt,  pick  a  speed  dude,  slow  or  fast,  I  prefer  fast  but  stay  with  it 

6m24s 

Driver 

You  want  to  stop  busting  my  balls? 

6m28s 

Crew  Comd 

If  you  would  maintain  a  certain  speed  I’d  be  a  happy  guy  and  I’d 
leave  you  alone 

6m34s 

Driver 

Hey  Sean,  Sgt.  Major  jacked  me  up  for  my  sideburns  again  today 

6m38s 

Crew  Comd 

Go  figure  (laughs),  go  figure,  go  figure 

6m44s 

Crew  Comd 

Alright  listen,  I  don’t  know  what  the  problem  is  today  Matt,  but 
let’s  NOT  slow  down  for  the  culverts,  this  is  a  big  18  tonne 
armoured  vehicle  and  it  can  take  the  culvert,  trust  me 

6m52s 

Driver 

Listen  if  you  want  to  drive  I’ll  pull  over  right  here  and  you  could 
hop  in 

6m56s 

Crew  Comd 

Yeah  like  you  always  say,  run  away  from  the  situation,  you’re  so 
soft 

7m00s 

Driver 

I’m  tired  of  your  nagging 
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Time 

Person  Talking 

Commands/Comms 

7m01s 

Crew  Comd 

Yeah  well  I’m  here  as  your  mother,  remember  that 

7ml3s 

Crew  Comd 

Does  that  look  like  12  up  there? 

7ml6s 

Driver 

Yeah  I  think  so,  yeah 

7m21s 

Crew  Comd 

Ah  Okay.  1 1  this  is  1 1  A,  be  advised  we’re  approaching  c/s  12 

7m28s 

Crew  Comd 

12  this  is  1 1  A,  be  advised  our  call  sign  will  be  passing  through 
your  location  momentarily,  we’ll  be  passing  to  your  left,  over 

7m35s 

PI  Comd  12 

12,  Roger,  we’ll  watch  the  flanks  as  you  pass 

7m46s 

Crew  Comd 

1 1 A  Roger,  out 

1  lm26s 

Crew  Comd 

Alright  guys,  just  to  let  you  know,  this  is  a  stretch  of  area  that  2 
commander  was  talking  about  with  the  4  IED  hits,  just  be 
vigilant,  it’s  about  a  kilometer  here  until  the  next  ANA 
checkpoint  where  we  can  turn  left 

1  lm38s 

Driver 

Don’t  worry  about  it  man,  I’m  wearing  my  lucky  underwear 

1  lm40s 

Crew  Comd 

Let’s  just  find  your  lucky  foot  so  you  can  pick  up  the  speed 

llm44s 

Driver 

Enough  of  the  jacking,  Jesus! 

llm47s 

Crew  Comd 

Would  you  just. .  .drive. .  .the  vehicle 

1  lm50s 

Driver 

I’m  the  one  who  signs  for  the  thing,  I’m  responsible  for  it 

1  lm53s 

Crew  Comd 

Who  signs  for  what?  Let’s  not  forget  who  (inaudible) 

llm56s 

Driver 

Yeah  but  you  don’t  have  to  do  the  maintenance  now  do  ya? 

1  lm57s 

Crew  Comd 

No  exactly,  that’s  your  job,  I  get  paid  the  big  bucks  to  make  sure 
you  do  it 

12m01s 

Crew  Comd 

Listen  Eric,  just  scan  left  and  right  to  make  sure  you  don’t  see 
anything  bad 

12m03s 

Gunner 

Kay 

12m06s 

Driver 

You  know,  Eric  you  could  back  me  up  sometimes  you  know. 

Why  am  I  always  the  one  getting  shit  on? 

12ml2s 

Crew  Comd 

The  reason  why  is  because  Eric  does  his  job  and  you  always 
complain  and  he  doesn’t. 

12ml5s 

(Inaudible  banter) 

12m23s 

Gunner 

Yeah  Matt,  drive  the  vehicle 

12m25s 

Driver 

Oh  yeah,  here  we  go,  they’ve  teamed  up  now 

14ml6s 

Crew  Comd 

Alright  that’s  that  an  ANA  checkpoint  coming  up  there  to  the 
right,  we’ll  be  turning  after  that 

14m23s 

Driver 

I  know  where  we’re  going 

14m27s 

Crew  Comd 

(laughing)  yeah  that’s  what  you  always  say,  that’s  why  when  we 
drive  past  the  place  we’re  supposed  to  turn  and  you’re  like, 

“What?  Maybe  if  you  told  me!” 
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Annex  C  L-681  protocol 


C.1  Executive  summary 

Protocol  L681 

Evaluation  of  the  usability  of  the  Environmental  Familiarization  and  Indicator  Training 
Tool 

Principal  investigator:  Matthew  Lamb  (DRDC  Toronto) 

Co-Investigators:  Dr.  Jocelyn  Keillor  (DRDC  Toronto),  Cpl  Philip  Mach  (DRDC  Toronto); 
Sgt  Dorothy  Wojtarowicz  (DRDC  Toronto) 

Work  breakdown  element:  12rr03 

Executive  summary 

Anecdotal  evidence  indicates  that  experienced  convoy  crews  develop  an  intuitive  sense  of 
potential  Improvised  Explosive  Device  (IED)  threats  in  unfamiliar  terrain.  Often  crews  will 
determine  that  an  IED  threat  is  present  even  if  there  does  not  appear  to  be  obvious  indicators  of 
that  threat.  The  Environment  Familiarization  and  Indicator  Trainer  (EFIT)  is  a  tool  that  is  being 
developed  by  DRDC  Toronto  in  an  attempt  to  train  this  intuitive  IED  detection  capability.  EFIT 
provides  cultural  familiarization  of  the  operational  terrain  by  exposing  troops  to  real  video  of 
convoy  operations  integrated  with  current  high  resolution  satellite  imagery  and  vector  data.  In  this 
study,  soldiers  will  be  asked  to  perform  a  series  of  tasks  using  the  EFIT  software  which  will 
highlight  any  interface  design  deficiencies  in  the  software. 

Risks  and  medical  coverage 

The  risks  in  this  study  are  considered  to  be  minimal.  No  medical  coverage  is  required  for  this 
study. 

Benefits  of  the  study 

The  study  will  help  improve  and  assist  in  further  developing  EFIT  for  its  use  by  the  Canadian 
Forces.  Participants  will  gain  a  better  understanding  of  potential  IED  indicators  and  become  more 
familiar  with  the  Afghan  culture  and  environment.  Ideally,  they  would  also  gain  an  improved 
ability  to  detect  IEDs  and  the  CF  will  gain  a  better  understanding  what  an  effective  software 
system  for  training  soldiers  in  IED  indicators  may  look  like. 

Related  studies:  L-560  Amendment  13,  L564 

C.2  Protocol  body 

Protocol  L681 

Evaluation  of  the  usability  of  the  Environmental  Familiarization  and  Indicator  Trainer 
Prototype 

Principal  Investigator:  Mathew  Lamb  (DRDC  Toronto) 
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Co-Investigators: 

—Dr.  Jocelyn  Keillor  (DRDC  Toronto) 

— Cpl  Philip  Mach  (DRDC  Toronto) 

-Sgt  Dorothy  Wojtarowicz  (DRDC  Toronto) 

Run  Director:  Cpl  Philip  Mach  (DRDC  Toronto) 

Work  breakdown  element:  12rr03 

ACRONYMS 

CF  Canadian  F  orces 

CFB  Canadian  Forces  Base 

C-IED  Counter-IED 

DRDC  Defence  Research  and  Development  Canada 

EFIT  Environmental  Familiarization  and  Indicator  Trainer 

IED  Improvised  Explosive  Device 

IPT  Individual  Pre -Deployment  Training 

PSTC  Peace  Support  Training  Centre 

TTP  Tactics,  Techniques  and  Procedures 

BACKGROUND 

Improvised  Explosive  Devices  (IEDs)  continue  to  pose  a  significant  threat  to  the  safety  of 
Canadian  Forces  (CF)  personnel  currently  deployed  in  combat  and  peace  support  missions, 
especially  in  Afghanistan.  A  recent  memo  from  the  Chief  of  Defence  Staff  states  that  “[IEDs] 
remain  the  single  most  dangerous  weapon  employed  by  insurgents  in  Afghanistan,  accounting  for 
the  majority  of  Canadian  combat  related  casualties”  (Chief  of  Defence  Staff,  2008).  Thus  efforts 
are  ongoing  throughout  the  CF  and  the  Department  of  National  Defence  in  general  to  mitigate  the 
IED  threat. 

Many  Counter-IED  efforts  focus  on  developing  new  technologies  to  detect,  avoid,  neutralize  or 
protect  personnel  from  IEDs.  However  despite  technological  advances,  detecting  IED  still 
depends  very  much  on  the  judgment  and  unaided  perceptual  skills  of  soldiers  (Zorpette,  2008). 
Accordingly,  the  CF  Counter-IED  Tactics,  Techniques  and  Procedures  (TTPs)  handbook 
(Department  of  National  Defence,  2006)  directs  personnel  to  monitor  their  environment  for  the 
presence  of  a  number  of  perceptual  cues  (“indicators”)  that  suggest  an  elevated  probability  of  the 
presence  of  IEDs. 

The  CF  are  interested  in  improving  soldiers’  ability  to  be  aware  and  recognize  such  indicators 
through  new  training  methods  to  supplement  the  traditional  PowerPoint,  classroom  training  that 
soldiers  currently  receive.  The  IED  Awareness  Training  project  (12rr03)  at  DRDC  Toronto  is 
supporting  the  CF’s  efforts  on  C-IED  training  by  developing  Environment  Familiarization  and 
Indicator  Trainer  (EFIT)  software  which  combines  real  video  of  Afghan  convoy  operations  with 
high  resolution  satellite  imagery  and  vector  data,  such  as  position  of  roads,  bridges,  culverts  and 
other  terrain  features,  inside  a  commercial  mapping  and  data  analysis  software  package  called 
ArcGIS  created  by  ESRI,  based  in  Redlands,  California.  The  CF  is  currently  interested  in 
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embedding  this  software  into  pre-deployment  training  and,  in  particular,  the  Individual  Pre¬ 
deployment  Training  (IPT)  course  which  is  run  out  of  Canadian  Forces  Base  (CFB)  Kingston. 

Interaction  design,  a  discipline  which  provide  guidelines  on  how  to  optimally  design  software 
interfaces  to  suit  the  environment  in  which  they  are  to  operate,  dictates  that  “identifying  usability 
and  user  experience  goals  is  essential  for  making  every  product  successful”  (Preece,  2002)  and 
that  an  understanding  of  users’  needs  should  be  reflected  in  the  design  of  the  software  (Preece, 
2002).  In  order  to  evaluate  whether  EFIT  reflects  an  understanding  of  user  needs  and,  thus, 
suitable  for  eventual  integration  in  pre-deployment  training,  user  testing  of  the  software  is 
required. 

OBJECTIVES 

The  primary  objective  of  this  study  is  to  determine  the  how  usable  EFIT  is  for  soldiers  required  to 
learn  about  the  Afghan  environment  and  IED  indicators.  Since  EFIT  is  still  in  its  developmental 
stages,  these  data  will  allow  for  EFIT  to  be  designed  in  a  manner  which  will  be  catered  for  use  by 
CF  members.  The  secondary  objective  of  this  study  is  to  further  evaluate  the  effectiveness  of 
EFIT  as  a  training  tool  in  order  to  demonstrate  that  EFIT  does,  in  fact,  facilitate  a  familiarization 
to  the  Afghan  environment  and  to  IED  indicators. 

METHOD 

Selection  of  Participants:  Approximately  10  male  and  female  soldiers,  meeting  the  standard 
personnel  requirements  for  the  CF  (active  duty  age  range  18  to  60),  meeting  standard  vision  and 
health  requirements,  who  are  currently  at  the  Peace  Support  Training  Centre  (PSTC)  will  take 
part  in  this  study  as  part  of  the  activities  for  their  pre -deployment  training. 

Procedure  and  Design:  The  approach  which  will  be  used  for  this  study  is  that  of  usability  testing. 
In  this  approach,  a  set  of  tasks  will  be  provided  to  participants  and  data  will  be  gathered  while 
participants  carry  out  the  tasks. 

10  trials  will  be  carried  out,  each  requiring  one  participant.  At  the  beginning  of  each  trial,  the 
participant  will  be  provided  with  a  brief  description  of  the  EFIT  system  (Figure  C.l)  and  asked  to 
read  the  information  sheet  and  sign  the  consent  form  (Appendix  A,  Appendix  B)  before  being 
given  a  training  task  to  train  participants  in  the  use  of  the  ArcGIS  system.  Participants  will  be 
given  4  tasks  to  perform  using  the  EFIT  system.  Each  task  will  be  designed  to  test  a  particular  set 
of  functions  in  the  EFIT,  a  sample  display  of  which  can  be  seen  below.  The  user  will  be  asked  to 
perform  the  following  tasks: 

•  The  first  task  will  involve  finding  and  watching  a  short  video  of  a  convoy  travelling  away 
from  the  Kandahar  Air  Field, 

•  The  second  task  will  involve  adjusting  the  options  of  the  video  viewed  in  Task  1  to  display 
more  information, 

•  The  third  task  will  ask  the  user  to  use  the  playback  features  to  navigate  to  a  specific  point  in 
a  second  video,  and 

•  The  fourth  will  ask  the  participant  to  use  the  information  displayed  on  the  map  to  navigate 
to  various  points  in  a  third  video. 
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The  videos  are  of  actual  scenery  (urban  and  rural)  recorded  from  a  moving  vehicle  in 
Afghanistan.  These  videos  depict  what  it  would  look  like,  from  the  point  of  someone  sitting  in  a 
vehicle,  to  transit  through  terrain  in  the  environments  that  CF  troops  patrol  and  are  required  to 
scan  for  IEDs  in  Afghanistan. 

While  the  participants  complete  these  tasks,  they  will  be  asked  to  follow  the  “think-aloud” 
protocol.  The  investigators  will  be  taking  written  notes,  as  well  as,  capturing  mouse  clicks  using 
Morae,  software  created  by  TechSmith  based  out  of  Okemos,  Michigan,  and  recording  the  user 
completing  the  task  using  video  cameras. 

Upon  completing  these  four  tasks,  participants  will  be  asked  to  fill  in  a  web-based  usability 
questionnaire  derived  from  the  work  of  Chin  et  al.  (1988,  Appendix  C),  as  well  as,  a 
questionnaire  prepared  for  use  in  this  particular  study  asking  users  what  the  desirability  of 
possible  additions  to  the  system  are  (Appendix  D). 


Figure  C.l:  Screenshot  ofEFIT. 


DATA  ANALYSIS 

The  data  will  be  used  to  evaluate  and  guide  further  development  and  design  of  the  EFIT  interface 
in  the  following  ways. 

The  mouse  click  data  will  be  used  to  identify  any  misleading  information  presented  on  the  screen, 
in  particular,  the  data  will  be  examined  to  identify  locations  on  the  screen  on  which  the  user 
clicked  which  may  reveal  interface  elements  that  do  not  meet  users’  interaction  expectations. 

The  video  data  will  act  as  supplement  to  the  written  notes  which  will  be  taken  by  the  observers. 
The  written  notes  will  capture  any  observable  frustration,  confusion  or  other  reactions  by 
participants  to  the  software  so  that  those  instances  can  be  later  reviewed  in  the  recorded  video  for 
further  analysis.  These  data  will  further  highlight  any  short  comings  of  the  EFIT  interface.  The 
video  will  also  allow  for  the  verification  that  the  users’  conceptual  model  of  EFIT  is  similar  or 
identical  to  our  own.  A  conceptual  model  is  how  a  system  is  expected  to  act  and  behave  (Preece 
2002)  and  to  ensure  proper  design  of  EFIT  it  is  critical  that  this  aspect  can  be  verified. 

Means  and  standard  deviations  will  be  calculated  and  compared  for  the  results  of  the 
questionnaire  found  in  Appendix  D  to  identify  desirable  functionality  and  the  usefulness  of  EFIT. 
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The  data  collected  from  both  questionnaires  will  be  used  as  a  baseline  for  comparison  in  future 
evaluations  of  EFIT. 


TIME  INVOLVEMENT 

Participation  in  this  study  will  require  1.5  to  2  hours  per  participant. 

REMUNERATION 

As  participants  will  be  taking  part  in  this  study  as  part  of  their  pre -deployment  training  at  the 
PSTC  they  will  thus  be  performing  their  normal  duties,  and  they  will  not  be  remunerated  for  their 
participation. 

MEDICAL  RISKS 

The  risks  in  this  study  are  the  same  that  CF  personnel  would  encounter  during  the  course  of  their 
normal  duties,  namely  those  involved  in  sustained  use  of  computer  equipment  (low  to  moderate 
eye  and  hand  strain)  and  the  stress  of  a  decision  making  task  varying  in  difficulty  from  low  to 
moderately  high.  The  videos  will  not  be  publicly  available,  yet  may  be  shown  in  future 
presentations  or  demonstrations.  In  the  event  that  a  video  from  this  trial  is  shown,  the  identity  of 
the  participant  will  be  protected. 

PHYSICIAN  COVERAGE 

This  study  is  considered  to  involve  minimal  risk  and  will  not  require  a  physician  to  be  present. 
There  will  be  no  medical  screening. 

BENEFITS  OF  THE  STUDY 

To  participants:  They  will  gain  a  better  understanding  of  potential  IED  indicators  and  become 
more  familiar  with  the  Afghan  culture  and  environment.  Ideally,  they  would  also  gain  an 
improved  ability  to  detect  IEDs. 

To  the  CF:  The  CF  will  gain  a  better  understanding  what  an  effective  software  system  for  training 
soldiers  in  IED  indicators  may  look  like. 
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C.3  Information  sheet 

Protocol  L-681 

Evaluation  of  the  usability  of  the  Environmental  Familiarization  and  Indicator  Training 
Tool 

Principal  Investigator:  Matthew  Lamb  (DRDC  Toronto) 

Co-Investigators:  Dr.  Jocelyn  Keillor  (DRDC  Toronto),  Cpl  Philip  Mach  (DRDC  Toronto);  Sgt 
Dorothy  Wojtarowicz  (DRDC  Toronto) 

Run  Director:  Cpl  Philip  Mach 

Experimental  protocol 

The  Canadian  Forces  (CF)  are  interested  in  improving  the  IED  threat  assessment  training  through 
the  use  of  video  captured  by  convoys  in  Afghanistan.  In  particular,  CF  personnel  have  expressed 
interest  in  embedding  the  Environmental  Familiarization  and  Indicator  Trainer  (EFIT)  software 
into  pre-deployment  training.  EFIT  is  a  software  system  that  is  currently  being  developed  by 
researchers  at  DRDC  Toronto  for  use  by  the  CF. 

In  this  study  you  will  be  asked  to  perform  a  series  of  tasks  using  the  EFIT  prototype.  Each  task  is 
designed  to  test  a  specific  set  of  functions  in  the  EFIT  system  and  are  not  meant  as  a  test  of  the 
IED  knowledge  that  you  possess.  You  will  be  asked  to  engage  in  “think-aloud.”  This  is  where 
you  say  what  you  are  thinking  in  terms  of  how  you  are  seeing  the  system,  what  you  think  various 
icons  or  pictures  on  the  screen  mean,  what  you  think  should  be  done  and  what  you  are  doing  at 
any  given  point.  This  again  is  to  further  our  knowledge  of  how  you  perceive  the  system  and  is  not 
an  indication  of  your  knowledge  and  will  have  no  bearing  on  your  performance  on  the  course 
which  you  are  taking.  The  purpose  of  this  study  is  to  test  how  easy  EFIT  is  to  use  and  not  your 
knowledge. 

After  completing  the  tasks  you  will  be  asked  to  fill  in  2  short  questionnaires.  The  first  one  will 
provide  us  with  your  thoughts  on  how  easy  EFIT  is  to  use  and  your  general  impression  of  the 
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system.  The  second  questionnaire  will  ask  you  to  rate  the  desirability  of  features  that  we  are 
considering  as  possible  additions  to  EFIT.  You  will  be  provided  with  a  description  of  the  each  of 
the  functions  so  that  you  can  properly  judge  their  desirability. 

Investigators  from  DRDC  Toronto  will  be  present,  collecting  data  while  you  are  performing  the 
tasks  and  will  be  recording  audio  and  video  for  future  analysis.  Your  confidentiality  will  be 
protected  by  the  investigators  and  your  personal  information  will  not  be  associated  with  any 
reports,  publications  or  presentations  that  arise  from  this  study. 

C.4  Consent  form 

Protocol  L-681 

Evaluation  of  the  usability  of  the  Environmental  Familiarization  and  Indicator  Training 
Tool 

Principal  Investigator:  Matthew  Lamb  (DRDC  Toronto) 

Co-Investigators:  Dr.  Jocelyn  Keillor  (DRDC  Toronto),  Cpl  Philip  Mach  (DRDC  Toronto);  Sgt 
Dorothy  Wojtarowicz  (DRDC  Toronto) 

Run  Director:  Cpl  Philip  Mach 

I  have  read  the  experimental  protocol  and  have  had  opportunity  to  ask  questions  of  the 
investigators.  All  of  my  questions  have  been  answered  to  my  satisfaction  and  I  have  received  a 
copy  of  this  protocol  and  consent  form.  However,  I  may  obtain  additional  information  by 
contacting  Matthew  Lamb  (matt.lamb@drdc-rddc.gc.ca,  416-635-2000  x3006)  or  Dr.  Jack 
Landolt,  Chair  of  the  DRDC  Human  Research  Ethics  Committee  (HREC)  (416-635-2120, 
j  ack.  landolt@drdc  -rddc .  gc .  ca) . 

I  have  been  told  that  I  will  be  performing  a  series  of  tasks  using  the  EFIT  system  and  that  the 
session  will  last  between  1  to  2  hours.  Data  on  my  performance  will  be  collected  via  observation 
as  well  as  audio  and  video  recordings,  and  I  will  be  asked  to  complete  2  questionnaires  at  the  end 
of  the  session.  I  understand  that  I  will  be  video-taped  and  that  these  videos  may  be  used  in  future 
presentations  and  demonstrations  and  that  my  identity  will  be  protected  if  these  videos  are  shown. 
I  will  also  be  asked  for  information  about  my  background  and  operational  experience  with  the 
CF.  This  is  a  minimal  risk  study.  Also,  I  acknowledge  that  my  participation  in  this  study,  or 
indeed  any  research,  may  involve  risks  that  are  unforeseen  by  DRDC. 

I  understand  that  I  am  considered  to  be  on  duty  for  disciplinary,  administrative,  and  Pension  Act 
purposes  during  my  participation  in  this  study.  I  understand  that  my  name  will  not  be  identified  or 
attached  in  any  manner  to  any  publication  arising  from  this  study.  Moreover,  should  it  be 
required,  I  agree  to  allow  the  experimental  data  to  be  reviewed  by  an  internal  or  external  audit 
committee  with  the  understanding  that  any  summary  information  resulting  from  such  a  review 
will  not  identify  me  personally. 

My  duty  status  has  no  effect  on  my  right  to  withdraw  my  participation  in  the  research  portion  of 
this  exercise,  refusing  to  give  consent  to  interviews  or  to  my  data  being  used  by  the  investigators 
in  this  study.  If  I  do  withdraw  my  consent,  I  will  not  be  allowed  to  take  part  in  the  study  and  any 
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data  that  has  been  gathered  relating  to  me  will  be  destroyed.  I  also  understand  that  the 
investigators  may  terminate  my  participation  at  any  time,  regardless  of  my  wishes. 

I  understand  that  I  am  not  entitled  to  any  financial  remuneration  for  my  participation  in  this  study. 

Secondary  Use  of  Data: 

(Please  circle)  I  Consent/Do  Not  Consent  to  the  use  of  this  study’s  experimental  data  involving 
me  in  unidentified  form  in  future  related  studies  provided  that  review  and  approval  have  been 
given  by  DRDC  HREC.  I  understand  that  I  will  be  given  a  copy  of  this  consent  form  so  that  I  may 
contact  any  of  the  individuals  mentioned  in  Paragraph  1  above. 


Participant’s  Name _ 

Signature _ Date _ 

Address _ Telephone  number 

Name  of  Witness _ 

Signature _ Date _ 

Commanding  Officer’s  Name _ 

Signature _ Date _ 


Principal  Investigator’s  Name _ 

Signature _ Date 


FOR  SUBJECT  ENQUIRY  IF  REQUIRED  : 

Should  I  have  any  questions  or  concerns  regarding  this  project  before,  during,  or  after 
participation,  I  understand  that  I  am  encouraged  to  contact  Defence  R&D  Canada  -  Toronto 
(DRDC  Toronto),  P.O.  Box  200,  1133  Sheppard  Avenue  West,  Toronto,  Ontario  M3M  3B9.  This 
contact  can  be  made  by  surface  mail  at  this  address  or  in  person,  by  phone  or  e-mail  to  any  of  the 
DRDC  Toronto  numbers  and  addresses  listed  below: 

Principal  Investigator: 

Matthew  Lamb,  (416)  635-2000  ext.  3003,  Matthew.Lamb@drdc-rddc. gc.ca 

Chair,  DRDC  Toronto  Human  Research  Ethics  Committee  (HREC): 

JackLandolt,  (416)  635-2120,  Jack.Landolt@drdc-rddc. gc.ca 
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C.5  Task  descriptions 

Introduction 

Thank  you  for  participating  in  the  user  testing  for  Environment  Familiarization  and  Indictor 
Trainer  (EFIT).  This  booklet  contains  the  tasks  that  you  will  be  asked  to  carry  out  for  this  trial. 
The  testing  consists  of  4  tasks  and  completion  of  all  4  will  take  between  1  to  2  hours.  You  may 
take  a  break  at  any  time  during  the  session  or  stop  the  session  entirely  if  necessary. 

We  would  like  to  emphasize  that  we  are  interested  in  assessing  the  usability  of  the  application, 
not  your  performance  on  the  tasks. 

When  carrying  out  the  tasks,  please  speak  aloud  your  thought  processes  (i.e.,  your  actions, 
intentions,  frustrations). 

Do  you  have  any  questions  before  you  begin? 

When  you  are  ready,  please  proceed  to  the  next  page. 

About  EFIT 

The  Environment  Familiarization  and  Indicator  Trainer  (EFIT)  is  a  tool  being  developed  by 
Defence  Research  and  Development  Canada  -  Toronto  as  part  of  the  Counter  Improvised 
Explosive  Device  Technology  Demonstration  Program  (C-IED  TDP).  The  primary  aim  of  EFIT  is 
to  aid  in  the  training  of  those  deploying  overseas  in  learning  about  Afghanistan  terrain  and  culture 
and  becoming  more  aware  of  indications  of  potential  IED  strikes.  EFIT  aims  to  train  soldiers  in 
these  things  through  the  display  of  real  video  of  convoy  operations  with  communications,  expert 
commentary  and  data  on  the  location  and  type  of  past  IED  strikes. 

The  Peace  Support  and  Training  Centre  was  given  a  demonstration  of  this  software  and  have 
invited  us  here  to  help  in  the  continued  development  of  this  software. 

Please  take  a  moment  to  read  the  EFIT  fact  sheet. 

Do  you  have  any  questions  before  we  proceed? 

EFIT  101 

Please  take  a  moment  to  familiarize  yourself  with  the  EFIT  interface  (screen).  Please  do  not  use 
the  hashed  features  (). 
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EFIT  Toolbar 


Interface  Components 

Interactive  map:  The  interactive  map  can  be  used  to  view  roads,  past  IED  incidents  and  select 
route  video  tracks  to  watch.  Roads  are  shown  on  the  map  using  green  lines,  past  IED  incidents  are 
shown  with  diamond  symbols  and  labeled  with  the  type  of  IED.  The  route  video  tracks  are  shown 
by  thick  blue  lines.  These  tracks  can  be  clicked  to  play  video  of  that  track. 


Route  Video  T racks 


lEDs 


Roads 


Route  video:  The  Route  Video  panel  allows  users  to  play,  stop,  pause  and  fast  forward  through 
the  Route  Video  Tracks. 

Options  panel:  The  Options  Panel  has  all  of  the  Route  Video  Tracks  that  are  loaded  into  EFIT. 
To  play  a  video  click  on  the  name  of  the  route  and  click  the  play  icon  on  the  Route  Video  panel. 
The  Options  Panel  also  has  options  to  select  whether  you  want  to  play  a  Familiarization  Video  of 
the  route  video  or  an  Indicator  Training  video.  The  Familiarization  Video  is  an  unedited  video 
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whereas  the  Indicator  Training  video  has  expert  commentary  about  the  route  and  highlighted 
potential  IED  indicators. 

EFIT  toolbar:  The  tools  that  you  will  use  to  operate  EFIT  are  located  on  the  EFIT  toolbar: 


^  Opens  the  EFIT  program. 

k  Allows  you  to  select  and  click  on  the  map.  While  watching  a  video,  you  may  click  on  a  part 
of  the  track  to  jump  to  that  location  in  the  video. 

Allows  you  open  incident  reports  linked  to  a  particular  IED  strike.  Select  the  tool  and  if  a 
location  turns  blue  (as  shown  below)  then  you  may  click  on  it  using  this  tool  to  open  the 
related  incident  report. 


Allows  you  to  pan  the  map.  You  may  also  use  your  middle  mouse  button.  Click  on  the  map 
with  your  middle  mouse  button  and  hold  it  down  while  dragging  the  map  to  the  desired 
location  before  releasing  the  mouse  button. 


(+1  Allows  you  to  zoom  in  on  the  map.  You  may  also  use  your  middle  scroll  wheel. 
Ql  Allows  you  to  zoom  out  on  the  map.  You  may  also  use  your  middle  scroll  wheel. 


Please  let  the  instructor  know  when  you  are  comfortable  with  the  interface  of  EFIT  and  ready  to 
begin  the  first  task. 
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Task  1 


You  will  be  deployed  to  Afghanistan  in  several  months  and  have  received  EFIT  as  part  of  your 
pre-deployment  training  package.  You  know  that  you  will  be  travelling  between  Kandahar  Air 
Field  (KAF)  and  Kandahar  City  frequently.  To  prepare  for  your  tour,  take  some  time  to  get  used 
to  what  the  route  looks  like.  It  is  expected  that  you  will  be  in  the  gunner  position  for  the  majority 
of  your  tour. 

Task  2 

The  IED  threat  in  Afghanistan  is  currently  high.  To  prepare  yourself  for  this  high  level  of  threat, 
learn  about  potential  IED  indicators  on  the  route  between  KAF  and  Kandahar  City. 

Task  3 

You  are  about  to  go  on  your  first  convoy  mission  through  Kandahar  City  and  you  want  to  learn 
more  about  the  types  of  IED  indicators  that  typically  occur  in  an  urban  environment.  Your  crew 
commander  tells  you  that  canalizing  from  vehicles  is  something  that  you  will  need  to  prepare  for. 
Fist  below  examples  of  canalization  on  the  route  through  Kandahar  City. 

Task  4 

You  have  been  tasked  to  be  an  air  sentry  on  a  convoy  travelling  between  MSG  and  PBSB  and  you 
want  to  be  aware  of  possible  IED  threats  on  that  route.  Find  out  what  the  most  common  type  of 
IED  used  on  this  route  was  for  2008.  Fist  below  the  key  messages  from  the  crew  commander  in 
the  instructional  video. 


1. 


2. 


3. 


4. 
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C.6  Usability  questionnaire 

Questionnaire  for  I&er  Interface  5  aristae  timi  fiJ  e : .v/C ;  ■’D oc mEaus'!- □  2-CHmd ?  &2 AS etrin^i/piua i±.‘I!totoop/c[ititiDELi:  gj . him 


Questionnaire  for  User  Interface  Satisfaction 

Based  am:  CMu,  JP..  Diehl.  VA,  Nonran.  KL  [19BE)  Devslopmatf  c?/ an  JnstnsBmt  Mttaziijing  User  SatiyOEfton  qf  ihe 
J&rmTi-Campfittir  Inltiiflice.  ACMCHTS3  Proceedings.  213-2  IE.  C1PS3  ACM  fAbstrajctl  Copying  without  fee  is  permitted 
provided  feat  the  copies  are  not  made  or  distrilmiEd  for  direct  corrmppcial  advantage,  and  credit  to  fee  source  is  given  ©19E5- 1993 
University  of  Maryland.  This  display  is  for  educational  uses  only.  C  onmemal  use  re  quires  a  lire  use  from  the  Office  ef 
Technology  C  oinmercialn  atiorc  (3&I)  403-2711  otciffiynHgil.ignd  edu.  fOULS  Home  Pag-1 1  jush  c» 


Please  rate  your  satisfaction  with  My  HomePage  Maker. 


■  Try  to  respond  to  all  the  items, 

■  For  items  that  are  not  applicable,  use:  NA 

■  Make  sure  these  fields  are  filled  in:  System:  Email  to: 

■  Add  a  comment  about  an  item  by  clicking  on  its  D  icon,  or  add  comment  fields  for  all  items  by  clicking 

on  Comment  All. 

■  To  mail  in  your  results,  chck  on:  Mail  Data 


System: 


Email  to: 


Optionally  provide  comments  and  your  email  address  in  the  box  . 
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rigid 

o 

o 

0 

c 

o 

o 

O 

c 

o 

o 

flexible 

© 

SCREEN 

0 

1 

2 

5 

4 

5 

6 

7 

S 

9 

NA 

7.  Reading  characters  on  the 

hard 

o 

o 

o 

o 

o 

o 

o 

o 

easy 

o 

screen  P 

S .  Highlighting  simplifies  task  D 

nor  at  all 

Q 

o 

o 

G 

o 

o 

a 

c 

o 

o 

very  nmch 

o 

9.  Organization  of  information  P 

confusing 

O 

o 

g 

C 

o 

o 

G 

0 

o 

o 

very  dear 

o 

10.  Sequence  of  screens  P 

confusing 

O 

o 

0 

0 

o 

Q 

Q 

0 

c> 

o 

very  dear 

o 

TERMINOLOGY  AND 

SYSTEM  INFORMATION 

0 

1 

2 

5 

4 

& 

6 

7 

S 

9 

NA 

11.  Use  of  terms  throughout 

iumsisrscti 

o 

o 

a 

0 

o 

o 

o 

0 

o 

o 

consistent 

o 

system  P 

12.  Terminology  related  to  task  P 

□ever 

o 

o 

Q 

0 

o 

Q 

Q 

0 

o 

© 

always 

o 

13.  Position  of  messages  on 

inconsistent 

o 

o 

G 

G 

o 

a 

G 

c 

o 

o 

consistent 

o 

screen  P 

14.  Prompts  for  input  P 

confusing 

o 

o 

G 

0 

o 

o 

O 

c 

© 

o 

dear 

o 

1  of  2 
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Questionnaire  for  User  Interface  Satisfaction 


§] e : ff/C :  D oc tanenis^ □  2  0  and £,cl!C  rj etrinE  pma cL'Desldx^/ques-ti cmcEj. hnu 


15.  Computer  informs  about  its 
progress  P 

16.  Error  messages  P 

LEARNING 

17.  Leanung  to  operate  the 
system  P 

IS.  Exploring  new  features,  by 
trial  and  error  □ 

IS.  F^memfoering  names  and  use 
of  c  rwnn  sands  D 

20.  Performing  tasks  is 
straightforward  P 

21.  Help  messages  on  the  screen 

22.  Supplemental  reference 
materials  P 

SYSTEM  CAPABILITIES 

23.  System  speed  P 

24.  System  reliability  P 

25.  System  tends  to  be  P 

26.  Correcting  your  mistakes  P 

27.  Designed  for  all  levels  of 
users  □ 


List  the  most  negative  aspect (s): 

1. 

2. 

3. 


□ever 

O 

o 

g 

0 

o 

unMpfuJ 

O 

o 

o 

0 

o 

0 

1 

2 

5 

4 

difficult 

o 

o 

0 

c 

o 

difficult 

o 

o 

o 

c 

o 

difficult 

o 

o 

c 

0 

o 

□ever 

o 

o 

G 

o 

o 

unMpfuJ 

o 

o 

G 

G 

o 

confusing 

o 

o 

o 

c 

o 

0 

1 

2 

5 

4 

too  slow 

o 

o 

0 

c 

Q 

unreliable 

o 

o 

D 

G 

o 

noisy 

o 

o 

G 

o 

o 

difficult 

o 

o 

o 

G 

o 

□ever 

o 

o 

0 

c 

o 

0 

1 

2 

5 

4 

o 

G 

G 

o 

o 

always 

G 

o 

o 

c 

o 

o 

iKl-fili 

O 

5 

6 

7 

s 

9 

NA 

o 

G 

c 

O 

o 

easy 

o 

o 

o 

o 

o 

o 

easy 

o 

Q 

o 

Q 

Q 

o 

easy 

o 

o 

G 

c 

D 

o 

always 

o 

o 

G 

G 

o 

G 

help M 

o 

G 

G 

c 

o 

G 

dear 

o 

5 

6 

7 

s 

9 

NA 

o 

o 

0 

o 

o 

fact  Enciugh 

o 

G 

o 

G 

o 

G 

reliable 

o 

Q 

■o 

o 

Q 

o 

qtti&t 

o 

G 

G 

c 

o 

G 

easy 

o 

Q 

o 

o 

o 

o 

always 

o 

5 

6 

7 

s 

9 

NA 

List  the  most  positive  aspeet(s): 

1. 

2. 

3.  " 


M  a  il  [Jala  C  ormtcni  All  Top  of  Form  |  return  TO  referring  pact 
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M/02/1009  12:09  PM 


50 


DRDC-RDDC-201 5-R009 


C.7  Usefulness  questionnaire 

Environment  Familiarization  and  Indicator  Trainer  (EFIT)  evaluation 


1 .  Please  provide  the  following  demographics.  Date: 

Rank: _  Trade/Classification: _  Years  in  service: _ 

Approx  number  of  times  you  have  attended  MAT  or  ETHAR: _ 

List  your  Operational  Experience: _ 


If  deployed  to  Afghanistan,  what  was  your  role? _ 

2.  Please  rate  the  usefulness  of  the  following  proposed  additions/improvements  to  EFIT. 
(1  =  not  at  all  useful;  5  =  extremely  useful) 


1 

2 

3 

4 

5 

Have  videos  of  the  same  route  taken  at  different  times  of  day 

Add  image  intensified  (II)  video 

Add  infrared  (IR)  video 

Add  details  of  past  IED  incidents  on  route  map 

Add  locations  of  OPFOR  activity  on  route  map 

Highlight  locations  of  canalizing  ground  on  route  map 

Remove  map 

Add  locations  of  pogo-points/waypoints/checkpoints/destinations  on  route  map 

Add  the  option  to  switch  between  a  map  mode  vs.  the  satellite  imagery 

Show  alternate  routes  to  destinations  on  route  map 

Indicate  the  date  and  time  of  day  the  video  was  taken 

Add  the  name  of  the  route  the  video  was  taken 

Make  the  map  interactive  so  that  routes  can  be  clicked  on  to  play  video  sequence 

Increase  size/length  of  video 

Different  camera  views  (2nd  vehicle  in  OOM,  rear  view,  gun  turret  view) 

3.  Please  rate  the  usefulness  of  EFIT  for  learning  the  following. 
(1  =  not  at  all  useful;  5  =  extremely  useful) 


1 

2 

3 

4 

5 

Becoming  familiar  with  routes 

Becoming  familiar  with  the  terrain  in  theatre 

Becoming  familiar  with  the  culture  and  what’s  normal  in  theatre 

Finding  IED  indicators 

Determining  how  threatening  a  situation  is 

Becoming  familiar  with  being  in  a  convoy 

Training  for  scanning  an  arc 

DRDC-RDDC-201 5-R009 


51 


4.  Did  you  learn  anything  new  while  observing  EFIT  that  was  not  included  in  the  training  you 
received?  If  so,  what  did  you  learn? 


5.  Additional  comments. 
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List  of  symbols/abbreviations/acronyms/initialisms 


ANP 

Afghan  National  Police 

AO 

Area  of  Operations 

CAF 

Canadian  Armed  Forces 

CER 

Forces  Combat  Engineer  Regiment 

CFB 

Canadian  Forces  Base 

Comms 

radio  communications 

DAODs 

Defence  Administrative  Orders  and  Directives 

DRDC 

Defence  Research  and  Development  Canada 

EFIT 

Environment  Familiarization  and  Indicator  Trainer 

ETHAR 

Explosive  Threat  and  Hazard  Awareness  and  Recognition 

GPS 

Global  Positioning  System 

HVAC 

Heating,  Ventilation,  and  Air  Conditioning 

IED 

Improvised  Explosive  Device 

KAF 

Kandahar  Airfield 

MAT 

Mine  Awareness  Training 

MSG 

Ma'Sum  Ghar 

NA 

Not  Applicable 

OOM 

Order  of  March 

OPFOR 

Opposing  Force 

PBSG 

Patrol  Base  Sperwan  Ghar 

PSTC 

Peace  Support  Training  Centre 

QUIS 

Questionnaire  for  User  Interface  Satisfaction 

R&D 

Research  &  Development 

recce 

Reconnaissance 

ROE 

Rules  of  Engagement 

SME 

Subject  Matter  Expert 

SUV 

Sport  Utility  Vehicle 

SVBIED 

Suicide  Vehicle  Borne  Improvised  Explosive  Device 

TDP 

Technology  Demonstration  Program 

TF 

Task  Force 

TMST 

Theatre  Mission  Specific  Training 

DRDC-RDDC-201 5-R009 
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TTPs 

VBIED 


Tactics,  Techniques  and  Procedures 
Vehicle  Borne  Improvised  Explosive  Device 
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1 3.  ABSTRACT  (A  brief  and  factual  summary  of  the  document.  It  may  also  appear  elsewhere  in  the  body  of  the  document  itself.  It  is  highly  desirable  that 
the  abstract  of  classified  documents  be  unclassified.  Each  paragraph  of  the  abstract  shall  begin  with  an  indication  of  the  security  classification  of  the 
information  in  the  paragraph  (unless  the  document  itself  is  unclassified)  represented  as  (S),  (C),  (R),  or  (U).  It  is  not  necessary  to  include  here  abstracts  in 
both  official  languages  unless  the  text  is  bilingual.) 

Defence  Research  and  Development  Canada  Toronto  Research  Centre  is  developing  the 
Environment  Familiarization  and  Indicator  Trainer  (EFIT)  as  part  of  the  Improvised  Explosive 
Device  (IED)  Awareness  Training  Project  of  the  Counter  Improvised  Explosive  Device 
Technology  Demonstration  Program.  EFIT  is  a  training  tool  that  provides  instruction  on  IED 
combat  indicators  and  what  to  look  for  in  a  given  area  of  operation. 

A  user  evaluation  was  conducted  with  a  prototype  of  Interactive  EFIT  software  with  seven 
members  of  the  Peace  Support  Training  Centre  in  Kingston  Ontario.  The  intent  of  the  evaluation 
was  to  determine  the  usability  of  Interactive  EFIT  and  to  gain  user  feedback  on  EFIT’s  training 
potential. 

Participants  were  briefed  on  how  to  use  the  software  and  they  were  asked  to  perform  four  tasks 
using  the  think-aloud  technique.  Observations  were  made  as  to  how  participants  interacted  with 
the  software.  Participants  also  completed  a  usability  questionnaire,  rated  the  usefulness  of 
Interactive  EFIT  for  seven  learning  objectives  and  rated  the  usefulness  of  proposed  features. 

Overall,  participants  evaluated  the  Interactive  EFIT  positively  and  found  it  to  have  potential  as  a 
training  tool.  Improvements  should  be  made  to  Interactive  EFIT  to  make  the  software  more 
flexible,  stimulating  and  intuitive. 


Le  Centre  de  recherche  de  Recherche  et  developpement  pour  la  defense  Canada  -  Toronto 
travaille  a  la  mise  au  point  d’un  simulateur  d’indices  et  de  familiarisation  avec  l’environnement 
(EFIT)  dans  le  cadre  du  projet  de  sensibilisation  aux  dispositifs  explosifs  de  circonstance  (IED) 
du  Programme  de  demonstration  de  la  technologie  pour  contrer  les  dispositifs  explosifs  de 
circonstance.  L’EFIT  est  un  outil  offrant  une  formation  sur  les  indices  de  combat  avec  IED  et 
expliquant  ce  qu’il  faut  tenter  de  reperer  dans  une  zone  d’ operations  donnee. 

Sept  membres  du  Centre  de  formation  pour  le  soutien  de  la  paix  de  Kingston  (Ontario)  ont 
evalue  un  prototype  du  logiciel  interactif  EFIT.  Cette  evaluation  par  des  utilisateurs  visait  a 
determiner  la  convivialite  du  logiciel  et  a  obtenir  des  commentaires  sur  son  potentiel  comme 
outil  de  formation. 

Les  participants  ont  ete  informes  sur  la  fagon  d’utiliser  le  logiciel  et  ont  ete  invites  a  accomplir 
quatre  taches  au  moyen  de  la  technique  de  reflexion  a  haute  voix.  On  a  ainsi  pu  observer 
comment  les  participants  utilisaient  le  logiciel.  Ceux-ci  ont  egalement  rempli  un  questionnaire 
sur  la  convivialite,  evalue  l’utilite  de  l’EFIT  interactif  relativement  a  sept  objectifs 
d’apprentissage  et  evalue  l’utilite  des  fonctionnalites  proposees. 

Dans  l’ensemble,  les  participants  ont  fait  une  evaluation  positive  du  simulateur  interactif  et 
estiment  qu’il  pourrait  constituer  un  bon  outil  de  formation.  Des  ameliorations  devraient  etre 
apportees  pour  creer  un  logiciel  plus  souple,  plus  stimulant  et  plus  intuitif. 
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